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Executive Summary 



The Comprehensive School Reform (CSR) program is one response to the persistent failure 
of some schools to provide students with educational opportunities to meet high standards for 
learning. The program was formed in an atmosphere of increased focus on school accountability 
and provides both a framework and the funding to enable schools to change their organization 
and practices so all students can achieve high standards. 

In 1998, Congress appropriated $145 million for the Comprehensive School Reform 
Demonstration (CSRD) program. It was designed not as an add-on to be placed on top of already 
existing programs and efforts but as a way to encourage schools to integrate local, state, and 
federal resources into a comprehensive effort that would better meet student learning needs. Like 
schoolwide Title I programs, CSRD was intended to help schools use multiple sources of funds 
and integrate programs while allowing flexibility and enhancing accountability for student 
learning. Its unique aspect was the expectation that schools would collaborate with expert 
partners to implement whole-school reform models that had a strong research base and a 
successful replication record. 

With the passage of the No Child Left Behind (NCLB) Act in 2002, CSR became a fully 
authorized program and is no longer a demonstration program. Further, NCLB described 1 1 
components of comprehensive school reform (Exhibit E-l), and, some argue, focused less on 
models than on the underlying processes that facilitate the kinds of changes needed in order for 
schools to ensure that all students learn. 




Exhibit E-l 

Eleven Components of Comprehensive School Reform 
Described in the No Child Left Behind Act 

■ Proven methods and strategics for student learning, teaching, and school management that are based 
on scientifically based research and effective practices and have been replicated successfully in 
schools with diverse characteristics. 

■ Comprehensive design for effective school functioning, integrating instruction, assessment, 
classroom management, and professional development and aligning these functions into a schoolwide 
reform plan designed to enable all students to meet challenging state content and performance 
standards and address needs identified through a school needs assessment. 

■ Professional development. High-quality and continuous teacher and staff professional development 
and training. 

■ Measurable goals for student performance and benchmarks for meeting those goals. 

■ Support from staff. Support from school faculty, administrators, and staff. 

■ Support for staff. Support for school faculty, administrators, and staff. (Added in 2001) 

■ Parent and community involvement. Meaningful involvement of parents and the local community 
in planning and implementing school improvement activities. 

■ External assistance. High-quality external support and assistance from a comprehensive school 
reform entity (which may be a university) with experience in schoolwide reform and improvement. 

■ Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

■ Coordination of resources. Identification of how other available resources (federal, state, local, or 
private) will help the school coordinate services to support and sustain the school reform. 

■ Scientifically based research. Scientifically based research to significantly improve the academic 
achievement of students participating in such programs as compared with students in schools who 
have not participated in such programs or strong evidence that such programs will significantly 
improve the academic achievement of participating children. (Added in 2001) 

Source: No Child Left Behind Act, Title I, Part F, Section 1606. 



Study Purpose 

The Longitudinal Assessment of Comprehensive School Reform Implementation and 
Outcomes (LACIO) responds to the NCLB Act’s requirement for an evaluation of the federal 
Comprehensive School Reform (CSR) program. The legislation stipulates two broad goals for 
the evaluation: first, to evaluate the implementation and outcomes achieved by schools after 
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three years of implementing comprehensive school reforms and, second, to assess the 
effectiveness of comprehensive school reform in schools with diverse characteristics. In order to 
address these requirements, the study focused on four evaluation questions: 

1. How are CSR funds being targeted? 

2. How is comprehensive school reform implemented in schools receiving CSR funds, 
in schools receiving Title I funds and in other schools? 

3. What is the relationship between CSR implementation and student achievement 
outcomes? 

4. What conditions (at the state and district level) influence the implementation of 
comprehensive reform programs? 

This report presents data collected from a random sample of 400 CSR schools that received 
funding in 2002 and 400 non-CSR schools with similar demographic and achievement 
characteristics. 1 It draws from three data sources — school-level surveys of principals and 
teachers, the National School-Level State Assessment Score Database and the National Center 
for Educational Statistics (NCES) Common Core of Data (CCD). The data were collected in 
spring 2003 at the end of the first year of CSR implementation (2002-03). The emphasis in this 
report is on the first two evaluation questions that focus on school reform activities and the 
targeting of CSR program funds. 



First- Year Findings 

The first year of the evaluation has yielded information with implications for federal 
policy. The implications relate to two key findings: 

• Although both CSR and non-CSR schools are engaged in reform, reform in 
CSR schools is more likely to include adoption of models and other activities 
closely associated with research-based models. 

• CSR funds are strongly targeted to high-poverty schools and low-performing 
schools, and schools receiving CSR funds are lower performing than are other 
schools with similar demographic characteristics at the time they receive 
awards. 



1 The sample of 400 represents 36 percent of the approximately 1,100 schools reported to receive CSR funds for the calendar 
year 2002. As a random sample, it does not mirror the universe on all characteristics. The distribution of the sample across locale 
and school level were comparable to the distributions of the universe, while reading and mathematics scores were slightly higher 
for the CSR sample. Similarly, the non-CSR schools, which were required to be in the same districts as the sample schools and 
with no current or past CSR funding, had slightly higher baseline achievement levels. These comparison schools represent the 
best available matches given these requirements. Further, analyses of achievement outcomes will control for variables such as 
achievement and poverty level, among others. 
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Implementation of School Reform Activities in CSR Schools and Other Schools 

Both CSR and non-CSR schools reported they were implementing specific activities that 
prior research indicates are associated with reform. However, as discussed below, the CSR 
schools differed from the non-CSR schools in their implementation of components directly 
related to selecting, implementing and evaluating models for reform. Further, CSR funding 
seems to contribute to building capacity for ongoing reform, with schools reporting more school 
activity that reflects coherence and cohesiveness during the first year of implementation (2002- 
OS) compared with the previous year. 

School Improvement Plans 

Nearly all schools in the sample (CSR and non-CSR schools) reported they had formal 
comprehensive plans for school reform. Principals at both CSR and non-CSR schools indicated 
these plans included components similar to the 1 1 CSR components, although CSR schools were 
more likely to report seeking research evidence about a proposed reform and adopting a reform 
created outside of the school. 

Teachers and school administrators were involved in selecting the reform model or 
approach being implemented, both at CSR schools and non-CSR schools. However, the 
school board and the district central office played a more significant role in selecting 
reform at non-CSR schools than at CSR schools, indicating more “top-down” requirements 
for changes in practice in non-CSR schools. One third of CSR schools reported that the district 
central office was one of several entities responsible for selecting the reform, compared with 57 
percent of non-CSR schools, which reported this method. Non-CSR schools reported that school 
board members were involved in the decision at a higher rate than did CSR schools (24 percent 
for non-CSR schools compared with 15 percent of CSR schools) (Exhibit E-2). Further, state or 
district mandates were more likely to contribute to the selection of reform at non-CSR schools 
(60 percent) than at CSR schools (31 percent). 
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Exhibit E-2 



Entities Involved in Selecting a Reform Model or Approach at the School 




*Difference is statistically significant at the .01 level. 



Exhibit reads: School staff were involved in selecting a reform model or approach at both 
CSR and non-CSR schools. However, the district central office (57 percent) and school 
boards (24 percent) had a greater role at non-CSR schools than at CSR schools (34 
percent; 15 percent). 

A higher percentage of CSR principals reported their schools had a comprehensive 
written plan in the first year of CSR implementation, 2002-03, (93 percent) as compared 
with the previous year (75 percent), indicating some influence of CSR. In contrast, principals 
in non-CSR schools reported little change (89 percent had comprehensive written plans in 2002- 
03, compared with 86 percent in 2001-02). In addition, CSR schools were significantly more 
likely to report engaging in whole school reform in 2002-03 (76 percent) than the prior year 
(55 percent). 

Professional Development 

Professional development for all teachers was included in the school reform plan 
more frequently in CSR schools than in non-CSR schools. A greater number of CSR 
schools than non-CSR schools provided more than 10 days for professional development 
and received on-site assistance from external supporters. 

Ninety percent of CSR schools included professional development for all teachers in their 
school reform plan compared with 73 percent of non-CSR schools. CSR schools also provided 
more than 10 days for professional development more often than did non-CSR schools (56 
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percent as compared with 39 percent). External assistance providers supported reform efforts on- 
site in significantly more CSR schools (86 percent) than non-CSR schools (57 percent). Finally, 
formal evaluation plans in CSR schools were more likely to include assessment of the utility of 
external assistance than such plans in non-CSR schools (41 percent vs. 30 percent) (Exhibit E-3). 



Exhibit E-3 

Status of Professional Development in School Reform 





CSR schools 


Non-CSR 

schools 


Professional development for all teachers is 
included in school improvement plan 


90% 


73 %* 


School provides 10 or more days for 
professional development 


56% 


39%* 


School receives on site support for reform 
efforts from external providers 


86% 


57%* 


School evaluation plan includes assessment of 
the utility of external assistance 


41% 


30% * 



*Dijference is statistically significant at the .01 level. 

Exhibit reads: In CSR schools as compared with non-CSR schools, professional 
development more often was included in the school reform plan (90 percent vs. 73 
percent), offered for over 10 days (56 percent vs. 39 percent) and took the form of on-site 
assistance from external sources (86 percent vs. 57 percent). 

Teacher participation in grade-level or content area teams increased significantly in CSR 
schools in 2002-03, compared with 2001-02. In addition, CSR teachers reported receiving more 
days of professional development in 2002-03 than in the prior year, and the training was more 
focused on issues related to reform (Exhibit E-4). 





Exhibit E-4 

Types of Professional Development in Which CSR Teachers Participated During the 
First Year of Reform Implementation (2002-03) Compared with the Previous Year 



100% 



80 % 



60 % 




40 % 



20% 



Reading/lang. Model Mathematics Intrepreting Consistency Instructional 

arts implementation reports with content strategies 



Monitoring Consistency with 
students' assessments 
progress 



□ 2001-02 ■ 2002-03 



Difference is statistically significant at the .01 level for all items. 



Exhibit reads: CSR teachers were much more likely to receive training on model 
implementation (85 percent), monitoring students’ progress (70 percent) and interpreting 
reports (83 percent) in 2002-03, compared with 2001-02 (the year prior to CSR 
implementation). 



Support for School Reform 

Significant differences existed between CSR and non-CSR schools in the type of support 
they receive. As might be predicted, CSR schools were far more likely to receive support 
from a model developer than non-CSR schools (31 percent vs. 6 percent). However, non-CSR 
schools were more likely to report receiving support for school reform efforts from the 
district than were CSR schools (72 percent of non-CSR schools vs. 34 percent of CSR schools) 
(Exhibit E-5). 
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Exhibit E-5 

Entity Primarily Responsible for Supporting Reform Efforts at the School 




*Dijference is statistically significant at the .01 level. 



Exhibit reads: CSR schools were far more likely to identify model developers as the 
primary supporters of reform at their school (31 percent for CSR schools vs. 6 percent for 
non-CSR schools). Conversely, non-CSR schools (72 percent) reported more district 
support for school reform efforts than did CSR schools (34 percent). 

States and districts were more likely to provide funds for reform to non-CSR schools 
than to CSR schools. Discretionary district funds went to 58 percent of non-CSR schools 
compared with 44 percent of CSR schools. Special state grants were awarded to 53 percent of 
non-CSR schools compared with 44 percent of CSR schools (Exhibit E-6). Further, in 2002-03, 
districts supported different kinds of activities in CSR schools as compared with the previous 
year. Districts were more likely to help CSR schools select a school reform model in 2002-03 
(45 percent) than in 2001-02 (32 percent) but were less likely to provide CSR schools with 
professional development for school reform in 2002-03 (72 percent) than in 2001-02 (86 
percent). 
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Exhibit E-6 

Sources of Funding that Contribute to Implementation 
and Operation of School Reform 



100% -i 
80 % - 



60 % 



40 % 



20% 



0% 



44 % 



58 % 









Discretionary district funds* 



53 % 



44 % 



Special state grants* 



■ CSR Schools □ Non-CSR Schools 



*Dijference is statistically significant at the .01 level. 



Exhibit reads: Special state grants and district discretionary funds were more commonly 
used for reform at non-CSR schools (58 percent for district funds; 53 percent for state 
funds) than at CSR schools (44 percent for each source of funds). 



Summary of CSR Implementation 

CSR comprises 11 components whose interaction may improve schools. Respondents to 
the survey indicated that both CSR and non-CSR schools were implementing a number of the 
components. However, CSR schools were more likely than non-CSR schools to implement 
components most associated with adopting a model. Consequently, the presences of some similar 
components in CSR and non-CSR schools may not indicate equal progress toward reform nor 
lead to equal outcomes for students. The differences in the components that are implemented in 
CSR and non-CSR schools may well encompass different interactions, which, in turn affect the 
extent to which schools are coherent and cohesive, enabling them to provide students with 
focused and challenging opportunities to learn to high standards. 

CSR schools, as compared with non-CSR schools, were more likely to implement the 
following components: 



• Adopt externally developed strategies that have been replicated. They did so 
by: 
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Identifying a specific reform model (85 percent compared with 49 
percent). 

Using evidence from research that the reform model chosen improves 
student achievement (42 percent compared with 26 percent). 

• Provide more continuous professional development. They did so by: 

Including professional development activities for all teachers (90 percent 
compared with 73 percent). 

Allocating over 10 days to teacher professional development (56 percent 
compared with 39 percent). 

• Include measurable goals for student performance associated with the reform 
model (57 percent compared with 41 percent). 

• Reflect support from staff by including a formal vote by teachers for the 
reform model (82 percent compared with 55 percent). 



• Provide support for staff by receiving on-site consulting relevant to the reform 
(85 percent compared with 57 percent). 

• Evaluate the reform. They did so by: 

Including the requirements of the reform model in the scope and content 
of evaluation (66 percent compared with 42 percent). 

Assessing the utility of external assistance (41 percent compared with 30 
percent). 

In sum, both CSR and non-CSR schools exhibited many aspects of comprehensive reform. 
However, CSR schools were more likely to adopt externally developed models. Other 
differences between the two types of schools were related to model adoption. 



Targeting of CSR Funds 

The legislation intends for CSR funds to be targeted to low-performing schools that serve 
high-need students. 

CSR funds were strongly targeted to high-poverty schools and those with high 
concentrations of minority students. Almost half (45 percent) of CSR schools had poverty 
rates of at least 75 percent, nearly three times greater than the percentage of all schools in this 
highest-poverty group (16 percent) and close to double the percentage of Title I schools (26 
percent) (Exhibit E-7). Similarly, schools with high concentrations of minority students (75 
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percent or higher) accounted for nearly half (47 percent) of CSR schools, compared with 30 
percent of Title I schools and 21 percent of all schools. CSR schools were much more likely to 
be located in urban areas (46 percent of CSR schools) than were Title I schools (26 percent) or 
all schools (25 percent). Rural schools were equally represented among CSR schools, Title I 
schools and all schools (13 percent of each group). CSR schools were less likely than Title I and 
all schools to be located in suburbs and towns. 

The distribution of CSR schools by poverty and minority status was similar to the 
distribution of Title I schoolwide programs — not a surprising finding, because both 
programs are targeted to high-need schools. For example, the highest-poverty schools 
accounted for 45 percent of CSR schools and 42 percent of Title I schoolwides. However, CSR 
schools were more likely to be located in urban areas (46 percent) than were Title I schoolwides 
(37 percent). The proportion of CSR schools that were operating Title I schoolwide programs 
was 56 percent, compared with 25 percent of all schools operating Title I schoolwide programs. 



Exhibit E-7 

Distribution of CSR Schools and Other Schools by School Poverty Rate 




CSR Schools Title I Schoolwide All Title I Schools All Schools 
Programs 



Exhibit reads: Almost half (45 percent) of CSR schools had poverty rates of at least 75 
percent, nearly three times greater than the percentage of all schools in this high poverty 
group (16 percent) and close to double the percentage of Title I schools (26 percent). The 
distribution of high poverty CSR schools was similar to the distribution of Title I 
schoolwide programs (42 percent). 

At the time of funding, CSR schools were significantly more likely to report that they 
were identified as a low-performing school according to the criteria used in their state (46 

percent) at the time of award than were the non-CSR schools (28 percent). When they received 
CSR funding they were also more likely to have received state sanctions due to low performance 
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(11 percent as compared with 3 percent). These are additional indicators that CSR funds are 
being targeted to schools in need of improvement. 



CSR schools had lower baseline achievement scores than did Title I schoolwides in 
reading and mathematics at most grade levels (elementary, middle, and high school) at the 
time the awards were made. For example, in elementary and middle grades, students in CSR 
schools scored an average of .4 standard deviations lower in reading and math achievement than 
students in Title I schoolwides. Further, the difference was true regardless of school locale or 
poverty level, a further indication of targeting. 



Conclusion 

Taken together, the findings about CSR implementation and targeting raise interesting 
questions. One such question is whether the use of CSR funds accelerates reform in the lowest 
performing schools. States and districts seem to have targeted CSR funds to those schools that 
have the greatest need to change practices in order to support high achievement for all students. 
With CSR funds, the schools were more likely to adopt models, focus professional development, 
and track student performance than were non-CSR schools. Both CSR and non-CSR schools 
were engaged in other reform activities. In subsequent years, the evaluation will provide 
information about whether CSR schools implement more reform components more thoroughly 
than do non-CSR schools. If they do, CSR can be seen as adding value to improvement by 
providing a mechanism that focuses efforts and enables school staff to organize themselves in 
ways that offer greater educational opportunities for students. Perhaps CSR helps schools jump- 
start improvement. 

Second, data from the first year of this evaluation indicate that all schools in the sample are 
engaged in many aspects of what the legislation defines as “comprehensive school reform.” 
Consequently, the study carries implications about the nature of reform in general. Most low- 
performing schools in the non-CSR group are making efforts to improve. Questions then arise as 
to whether the efforts are associated with improved outcomes: Do schools succeed in reform 
without models to organize them? Are models only important in the lowest performing schools? 
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I. Introduction 



This report of the Longitudinal Assessment of Comprehensive School Reform 
Implementation and Outcomes (LACIO) responds to the No Child Left Behind (NCLB) Act’s 
requirement that an evaluation of the federal Comprehensive School Reform (CSR) program be 
completed. The legislation stipulates two broad goals for the evaluation: 

• To evaluate the implementation and results achieved by schools after three 
years of implementing comprehensive school reforms. 

• To assess the effectiveness of comprehensive school reform in schools with 
diverse characteristics. 

The federal CSR program provides funds to states, which, in turn make grants to schools to 
support comprehensive reform. The intention is that the vast majority of these schools will be 
Title I schools “in need of substantially improving” their student achievement levels. Further, the 
CSR program delineates 1 1 components of “comprehensive school reform,” which are supported 
by research and evaluation. 

This first-year report begins with an overview of the context for the CSR program, a 
description of its history, and a description of the 1 1 components of comprehensive school 
reform included in NCLB. It then addresses the goals of NCLB by presenting preliminary 
findings related to implementation of CSR and an examination of the types of schools receiving 
CSR program funding. The report includes information drawn from surveys sent to 400 CSR 
program schools (“CSR schools”) and 400 matched non-CSR program schools (“non-CSR 
schools”) that examine the presence and characteristics of the 1 1 components of CSR, as well as 
other elements (e.g., school organization) that prior research has shown to be associated with 
successful program implementation. 



Background 

Comprehensive school reform was a response to the persistent failure of some schools to 
provide students with educational opportunities to meet high standards. As state and federal 
governments have increased emphasis on academic standards, they have also increasingly held 
schools accountable for ensuring that students meet those standards. However, they have also 
provided guidance and funds to help schools change so students can successfully achieve high 
standards. CSR constitutes one mechanism for providing such assistance. 

This section provides background on CSR, including the relationship of the program to 
ongoing efforts to hold schools accountable for results and the history of CSR as a program 
funded through the U.S. Department of Education (ED). It then moves to a discussion of the 
purpose of this evaluation, including how the study addresses emerging issues in CSR and the 
evaluation questions from NCLB. The section concludes with an overview of the report. 
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Assessment, Accountability, and Schoolwicle Reform 

Since 1965, the federal government has authorized formula grants to states and local 
education agencies (LEAs) for the education of elementary and secondary students with low 
academic achievement who are enrolled in schools serving low-income areas. These grants, 
known as Title I, were designed to accomplish four primary goals: 

• Provide supplemental education to students eligible for services. 

• Provide additional funding to schools and LEAs serving high concentrations 
of children from low-income fa mi lies. 

• Focus educators on the needs of special student populations. 

• Improve the academic achievement of eligible students, reduce performance 
gaps between advantaged and disadvantaged students, and assist eligible 
students in meeting high academic standards. 

In 1994, Congress changed the focus of Title I programs in the Improving America’s 
Schools Act (IAS A). This act, which reauthorized the Elementary and Secondary Education Act 
(ESEA), included Title I provisions calling for schools that receive Title I funds to set high 
standards for all students, to assess all students relative to these standards, to report results to the 
public, and to make instructional and structural changes to ensure that cdl students have the 
opportunity to meet these standards (Quenemoen et al. 2001). This movement, part of standards- 
based reform, marked a shift away from providing disadvantaged students with basic skills and 
toward more advanced content and performance standards for all students. 

The authorization signaled a new focus on schoolwide reform for Title I schools serving 
high concentrations of low-performing, high-poverty students (U.S. Department of Education 
2001). This focus came as evaluations suggested that targeted, “pull-out” education programs for 
students, the previous use of Title I funds, showed no clear positive effect on student 
achievement in high poverty schools. In fact, studies showed that pulling students out of their 
regular classes for special programs disrupted the classroom, stigmatized the students, reduced 
time spent in the regular class with their peers, and yielded uneven instruction (U.S. Department 
of Education 1997). In contrast, studies supported the notion that schoolwide reform would 
benefit even the most low achieving students by raising standards, implementing a challenging 
curriculum, and assessing learning (U.S. Department of Education 1996a; U.S. Department of 
Education 1996b; U.S Department of Education 1993). 

As a result, schools with high concentrations of high-poverty students were allowed to pool 
resources and encouraged to use these resources to leverage additional funds, as well as to 
integrate programs related to curriculum, parent involvement, professional development, and 
drug prevention (U.S. Department of Education 1998). The U.S. Department of Education 
identified several characteristics of schoolwide programs including: 
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• A comprehensive approach that integrates the whole school (students, faculty, 
parents, and the community), uses data to assess students’ needs, and then ties 
instructional and assessment practices in all curricular areas to this 
understanding. 

• A focus on examining and reforming the curriculum in multiple subject areas, 
not simply one or a few. 

• Collaboration between the school and district to implement reform, where the 
school receives autonomy in areas such as management, budget, and program 
development, while also getting district support and funding. 

• Strong leadership from the principal to shape a common vision. 

• Qualified professionals who receive professional development, small classes, 
and the right materials and equipment to facilitate excellent teaching. 

• An environment where everyone believes in the ability of students to achieve 
high standards — with no exceptions. 

• Accountability measures that monitor student progress, use data to 
continuously improve teaching and learning, and provide the necessary 
support for success (U.S. Department of Education 1998). 

Of course, change or reform is a complicated, demanding process. For change to take hold, 
consistent leadership is needed, as is support from the district (Finnan 2000; U.S. Department of 
Education 2000a; Stringfield et al. 1997). School reform is a political process requiring buy-in 
from a broad range of constituencies, including teachers, parents, and the larger community; 
developing this buy-in takes time. These challenges to reform are further complicated by 
changing expectations for schools, the growth of new programs, and the dismantling of old 
ones — all of which take staff time and can create a culture of cynicism about change (Sarason 
1996). 

NCFB raises the stakes for schools, particularly low-performing schools. As with earlier 
Title I authorizations, schools are required to make “adequate yearly progress” (AYP) toward 
state achievement standards. In addition, NCFB tightens requirements about how such progress 
is shown by mandating annual testing and reporting outcomes of key subgroups of students. 
Further, Title I schools “in need of substantially improving” student achievement levels can be 
subject to sanctions. Such schools also must make supplemental services available to students 
and can, if the need for improvement persists, be reconstituted. Students can also receive 
opportunities to attend different schools. 

The stakes, therefore, are high. Schools must focus on student achievement and change 
curriculum, instruction, organization, and other elements to meet students' needs better. CSR is a 
source of both ideas and funds to bring about school reform designed to facilitate schools’ 
abilities to meet accountability requirements. 



3 




The Comprehensive School Reform (CSR) Program 

In 1998, Congress appropriated $145 million for the Comprehensive School Reform 
Demonstration program (CSRD). It was designed as a way to encourage schools to engage in a 
comprehensive effort that would better meet student learning needs (U.S. Department of 
Education 2000a). CSRD was not to be an add-on placed on top of already existing programs 
and efforts but a way to encourage schools to integrate local, state, and federal resources to bring 
about improved student learning (U.S. Department of Education 1999). 

Like schoolwide Title I programs, CSRD was intended to help schools leverage funds from 
both public and private sources and integrate programs while giving them flexibility and 
enhancing accountability for student learning. Its unique aspect, relative to other Title I programs 
and the IASA legislation, was the expectation that schools collaborate with expert partners to 
implement whole-school reform programs that had a strong research base and a successful 
replication record (Hale 2000). 

As its cornerstone, CSRD had nine criteria that the reform programs used by funded 
schools had to meet (U.S. Department of Education 1999; U.S. Congress 105th Session). The 
legislation offered 17 programs as examples of the models schools might choose to employ but 
enabled schools to choose other models, combine models, or create their own reform programs, 
provided they met the nine criteria. The models include some developed for a school’s entire 
curriculum, some models focused on specific content areas, and process models that guide 
schools through the development of their own vision and corresponding materials and practices. 
In addition, schools had the option of crafting their own models (Hale 2000). A quick review of 
the models adopted by schools in the first cycle of CSRD funding indicates that subject-specific 
(mainly reading) models were among the "top 30" models adopted by schools 
(http://www.sedl.org/csrd/awards.html). Further, schools implemented different configurations 
of the nine components, with some schools focusing on fewer than all nine. 

The CSRD appropriation spurred a dramatic growth in school reform. Even prior to CSRD, 
more than 2,100 schools were affiliated with one of three schoolwide reform programs (Success 
for All, School Development Project, or Accelerated Schools) (Consortium for Policy Research 
in Education 1998). The CSRD initiative was expected to more than double the number of 
schools embarking on such reform efforts (Consortium for Policy Research in Education 1998). 
As of September 2000, 1,800 schools had received CSRD funds (U.S. Department of Education 
2000b). About 2,000 schools were funded as a result of the next round of applications for CSRD 
funds. Several states have adopted initiatives similar to CSRD. States such as Colorado, Hawaii, 
Wisconsin, and North Carolina have used the CSRD model to restructure their efforts at reform, 
providing similar grants to districts in their states; other states (Oregon, Tennessee, and West 
Virginia) used the CSRD model to guide how they distribute Title I and state school 
improvement funds (U.S. Department of Education 2000b). 

With the passage of the No Child Left Behind (NCLB) Act in 2001, CSR became a fully 
authorized program and is no longer considered a demonstration program. Further, NCLB 
described 11 components of comprehensive school reform (Exhibit 1) and did not include a list 
of models. 
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Exhibit 1 

Eleven Components of Comprehensive School Reform 
Described in the No Child Left Behind Act 

■ Proven methods and strategies for student learning, teaching, and school management that are based 
on scientifically based research and effective practices, and have been replicated successfully in 
schools with diverse characteristics. 

■ Comprehensive design for effective school functioning, integrating instruction, assessment, 
classroom management, and professional development and aligning these functions into a schoolwide 
reform plan designed to enable all students to meet challenging state content and performance 
standards and address needs identified through a school needs assessment. 

■ Professional development. High-quality and continuous teacher and staff professional development 
and training. 

■ Measurable goals for student performance and benchmarks for meeting those goals. 

■ Support from staff. Support from school faculty, administrators, and staff. 

■ Support for staff. Support for school faculty, administrators, and staff. (Added in 2001) 

■ Parent and community involvement. Meaningful involvement of parents and the local community 
in planning and implementing school improvement activities. 

■ External assistance. High-quality external support and assistance from a comprehensive school 
reform entity (which may be a university) with experience in schoolwide reform and improvement. 

■ Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

■ Coordination of resources. Identification of how other available resources (federal, state, local, or 
private) will help the school coordinate services to support and sustain the school reform. 

■ Scientifically based research. Scientifically based research to significantly improve the academic 
achievement of students participating in such programs as compared with students in schools who 
have not participated in such programs; or strong evidence that such programs will significantly 
improve the academic achievement of participating children. (Added in 2001) 

Source: No Child Left Behind Act, Title I, Part F, Section 1606. 

The schools included in this evaluation received funding starting in 2002. States applied the 

NCLB definition of comprehensive reform to the schools they funded as described in the 

Department's guidance for the program, despite the fact that some funding came from earlier 

appropriations. The 11 components, then, frame the study. The underlying questions related to 

implementation are: 
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• To what extent do schools receiving CSR funding implement the 1 1 
components? 

• How is implementation of reform in CSR schools different in schools, 
particularly Title I schoolwides, that do not receive program funding? 

This report focuses on the implementation question in the first year of funding. Later 
reports will assess progress on implementing the components and will relate implementation to 
outcomes in both CSR and non-CSR schools. 



Study Purpose 

The CSR program has evolved along with the changing context for education. It is one 
approach to improving opportunities for students in low-performing schools. NCLB includes 
other approaches, some of which focus on students (e.g., access to supplemental services; the 
option to transfer students to higher performing schools) and some of which focus on improving 
the schools (e.g., schoolwide Title I; CSR; professional development). Further, federal, state, and 
district policies create increased pressure on low-performing schools because accountability for 
results has increased. Such pressure could potentially increase “CSR-like” activities in non-CSR 
schools. As a result, the study, while focused on CSR, has implications for how all schools in 
need of improvement may serve their students better. This section focuses first on the potential 
implications of the evaluation and then moves to the questions that guided the evaluation, 
placing them within the broader context as well. 



Emerging Issues in CSR 

CSR as a program facilitates access to models with scientifically based evidence of their 
effectiveness. It also articulates a set of principles (in the form of components) that are designed 
to reform low-performing schools so students in such schools can meet high standards. The 
current study is evaluating the extent to which CSR achieves its objectives. As such, the findings 
of the study will have broad implications for both policy and practice. This section points to a 
few such implications and places this evaluation within the framework of earlier CSR-related 
research and evaluation related. 

This study represents an important change in evaluating CSR. Whereas earlier evaluations 
focused strongly on models, this evaluation focuses on comprehensive school reform as 
manifested in the 1 1 components and interactions among them. Examples of work focused on 
models abound. Borman, Hewes, Overman, and Brown used meta-analysis to review “research 
on the achievement effects of comprehensive school reform (CSR) and summarizes the specific 
effects of 29 widely implemented models” (Borman et al. 2003). Similarly, Desimone addressed 
the question “Can comprehensive school reform models be successfully implemented?” 
(Desimone 2002). Both articles focus attention on the use of “proven strategies and proven 
methods for student learning, teaching and school management” that have “been found to have 
strong evidence that such programs will significantly improve the academic achievement of 
participating children” (components 1 and 11). They also include some discussion of a 
“comprehensive design for effective school functioning,” “high quality and continuous teacher 
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and staff professional development,” and support by “teachers, principals and administrators” 
(components 2, 3, and 5). 



As NCLB makes clear, the use of scientifically based research is not confined to adopting 
models. The principle of basing practice on scientific research runs throughout the law. 
However, CSR focuses on principles beyond a single subject or service to students. CSR 
presumes that low-performing schools, as institutions, should (and can) change to serve students 
more effectively. The institutional focus of CSR differentiates it from, for example, programs 
that focus on improving reading instruction or any other curriculum area. Further, CSR assumes 
that comprehensiveness itself is a spur to reform. Although NCLB includes 1 1 components of 
comprehensiveness, their nature is such that interactions among them (and with the context in 
which the school exists) are expected to lead to greater impact than each of them alone or even a 
subset of them. In short, the CSR components are not a checklist for comprehensiveness, but 
rather are indicators of coherence and cohesiveness of school structures and processes 
(Newmann et al. 2001). This evaluation provides a vehicle for addressing such views 
empirically. 

As a result of the evaluation, ED will have information about the extent to which CSR 
helps low-performing schools change practices and improve outcomes with appropriate support, 
including support in implementing research-based practices and organizational structures. 



Evaluation Questions 

In NCLB, Congress required a national evaluation to: (1) evaluate the implementation and 
results achieved by schools after three years of implementing comprehensive school reforms; and 
(2) assess the effectiveness of comprehensive school reforms in schools with diverse 
characteristics (No Child Left Behind Act of 2001, P.L., 107-110). In order to address these 
requirements, this study is focused on four broad questions: 

1. How are CSR funds being targeted? 

Are states targeting CSR funds to low-performing and under-performing 
schools in both urban and rural areas? Does the funding reach students at 
all grade levels? 

2. How is comprehensive school reform implemented in schools receiving CSR funds, 
in schools receiving Title I funds, and in other schools? 

How well have schools implemented the eleven components of 
comprehensive school reform identified in the NCLB Act of 2001? 

What types of school reform models or strategies are schools 
implementing? 

Do CSR resources help schools build their capacity for reform? How do 
schools sustain the reform process, given natural attrition in staff, changes 
in district priorities, and shifts in funding? 
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3. What is the relationship between CSR implementation and student achievement 
outcomes? 

To what extent have CSR schools made progress on state assessments, in 
comparison to other schools in their state with similar characteristics? 

Were schools previously identified as in need of improvement able to 
make sufficient progress to move out of “school improvement” status? 

How do achievement trends vary across different types of CSR schools 
(e.g., high poverty, high minority)? 

4. What conditions (at the state and district level) influence the implementation of 
comprehensive reform programs? 

To what extent have state and district policies supported the 
implementation of comprehensive school reform? Have state and district 
policies or support helped develop the capacity to begin the reform 
process? How have states, districts, and schools planned to sustain 
reforms after the federal funding ends? 



Focus and Organization of the Report 

In this report, the primary emphasis is on the first two evaluation questions that focus on 
school reform activities and the targeting of CSR program funds. In the report, this chapter is 
followed by a description of the evaluation design, including data collection and analysis 
approaches. The third chapter presents the findings from the study, including: 

• A look at which schools received CSR program funds in 2002, including 
comparisons of baseline achievement levels of CSR schools and non-CSR 
schools prior to implementation. 

• The status of various aspects of reform in the CSR and non-CSR schools. 

• The school-based practitioners' perceptions of state and local policy influence 
in implementing school reform. 

The findings section concludes with a discussion of the nature of comprehensive reform as 
it currently exists in the sample, including how components relate to one another and the value of 
CSR funds in advancing reform and helping schools build their capacity for reform. The final 
chapter summarizes the findings and points to policy implications. 
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II. Evaluation Design 



The five-year evaluation will employ a quasi-experimental design to analyze student data 
and school reform at multiple points in time, comparing CSR program and non-CSR program 
schools. The evaluation will include multiple methods of data collection and analysis in order to 
increase the robustness of the study. The study will include three approaches, each contributing 
to answering the questions posed by the legislation. The approaches comprise a complementary 
set of inquiries, going beyond the capability of any single method. The approaches are: 

• An analysis of student achievement in all CSR schools that received funding 
in 2002, compared with student achievement in a sample of similar schools 
that have not received CSR funding as well as with a sample of Title I 
schoolwides. 

• An analysis of survey and interview data from a large sample of schools, a 
smaller sample of districts and each state. 

• A field-based inquiry of a small sample of CSR program and nonprogram 
schools in the district and state context. 

The section below briefly outlines the data collection and analysis plan for the five-year 
evaluation, as well as the evaluation activities used in preparing this first-year report. 



Study Data Collection and Analysis 



Analysis of Student Achievement 

For all schools in the evaluation, a quantitative analysis of changes in student achievement 
will be conducted. The analyses will cover the universe of CSR schools funded in 2002 
compared with a sample of 400 non-CSR schools selected to match the CSR schools in the 
sample for the surveys (see below). In addition, student achievement will be compared with a 
sample of Title I schoolwides. 



Surx’ey of School Reform Activities 

The study includes a survey of a sample of 800 schools (400 CSR schools and 400 
comparable schools) that will complete survey forms asking them to describe the various reform 
activities occurring at the schools. The CSR schools in the evaluation represent a random sample 
of all schools receiving CSR funds in 2002. 

The survey instrument will be distinguished from a typical survey form in that it will 
include items that are evidence-based and measure behaviors, rather than measuring attitudes and 
expectations (see Appendix B for a complete discussion of the use of the survey forms). The 
survey will inquire into the extent to which the 1 1 elements of comprehensive school reform 
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included in NCLB are present and other elements (e.g., school organization) that prior research 
shows to be associated with successful program implementation. 

In addition to the mailed surveys, the evaluation will collect data about CSR activities and 
state and district policies from a smaller sample of districts and each state. These telephone 
interviews will provide a more complete picture of CSR implementation, as well as data needed 
to analyze the effects of state and district policies on CSR implementation and outcomes. Such 
effects will be addressed using multilevel modeling techniques (Bryk and Raudenbush 1992). 
Subsequent reports will include those analyses. 



Field-Based Study of Reform Activities 

Out of the larger sample of 800 schools, a subsample of 15 pairs of schools (half of each 
pair is CSR program school and half, non-CSR) have been selected to participate in the 
Field-Based Study of Reform Activities at CSR Program and non-CSR Program Schools. 

The field-based study will include two visits to each site, occurring during the second and 
third years of a school’s CSR award. Each “site” consists of four entities: 

1. A CSR-funded school. 

2. A demographically matched non-CSR school (a school that has not received any 
federal CSR funds) located in the same district as the CSR-funded school. 

3. The district within which the two schools are located. 

4. The state within which the district is located. 

By covering these four entities, the field-based component will address all four evaluation 
questions and also produce an understanding of the dynamic of the actual relationships among 
school, district, and state actions, policies, and practices. 

During the site visits in the field-based component, the evaluation team will complete 
organizational inventories covering relevant events at the “site.” Data for the inventories will 
come from classroom observations, using a formal observation instrument, as well as other direct 
field observations, reviews of relevant school documents and materials, and discussions with 
school staff and parents. The field-based study is underway and will be concluded by early 
spring 2005. Data from the study will be included in later reports. 



Methods Used in the First-Year Report 

The First- Year Report presents data collected from a random sample of 400 CSR schools 
that received funding in 2002 and 400 non-CSR schools with similar demographic and 
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achievement characteristics. 2 It draws from three data sources — the mailed school-level surveys 
of principals and teachers, the National State Assessment Score Database and the National 
Center for Educational Statistics (NCES) Common Core of Data (CCD). 



Mailed School-level Survey 

Survey data were collected from a random sample of CSR schools that received funding in 
2002 and the comparison schools in spring 2003 at the end of the first year of implementation. 
Of the schools selected for the survey, 367 CSR and 356 non-CSR schools agreed to participate 
in the evaluation, with 350 of them representing matched pairs. In addition to schools that 
refused to participate, some late in the school year, the sample included five schools that had 
closed. They were replaced with a second random sample and mailed surveys in fall 2003 
(Appendix B). Five additional comparison schools were eliminated from the sample because 
they were later found to be CSR schools. These comparisons will not be replaced. 

The analyses presented in this year-one report are based on responses from 89 percent of 
participating target schools and 89 percent of participating non-CSR schools. Matching 
individual CSR and non-CSR schools is not crucial for the descriptive analyses in this report. It 
will become important when implementation and outcomes are related in subsequent years. 
Currently, the study data includes matched responses from 88 percent of schools. 



National State Assessment Score Database 

The National State Assessment Score Database , maintained by the American Institutes for 
Research (AIR), includes student achievement data from all states. The data will be used to 
compare outcomes in CSR and similar non-CSR schools. In the first-year report, the data were 
used to establish the baseline for later comparisons. 



Common Core of Data 

The CCD, which provides information about demographic and other school characteristics, 
served two roles in the first-year report. First, it was an important source of data for matching 
CSR and non-CSR schools. Second, CCD provided the data to analyze the extent to which CSR 
funds were targeted as intended by NCLB. 
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z The sample of 400 represents 36 percent of the approximately 1,100 schools reported to receive CSR funds for the calendar 
year 2002. As a random sample, it does not mirror the universe on all characteristics. The distribution of the sample across locale 
and school level were comparable to the distributions of the universe, while reading and mathematics scores were slightly higher 
for the CSR sample. Similarly, the non-CSR schools, which were required to be in the same districts as the sample schools and 
with no current or past CSR funding, had slightly higher baseline achievement levels. These comparison schools represent the 
best available matches given these requirements. Further, analyses of achievement outcomes will control for variables such as 
achievement and poverty level, among others. 
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III. First- Year Findings 



Comprehensive school reform is both a concept and a program. As a concept, it implies 
systematic, whole school, coherent reform in how a school is organized, delivers instruction, 
uses curriculum, provides professional development, and integrates resources. As a program, 
comprehensive school reform is defined by 1 1 components, which reflect research and practice 
in school reform. This evaluation is designed to provide information about both the concept of 
comprehensive school reform and the program, and the analyses will relate outcomes to school 
reform, not to program funding. Consequently, the data collection instruments included questions 
about the extent to which CSR and non-CSR schools were engaged in reform, whether the 
characteristics of reform differed in CSR and non-CSR schools, and the outcomes of reform 
activities in both types of schools. To that end, and because CSR is frequently associated with 
implementation of “models,” the surveys avoided the use of the terms “comprehensive school 
reform” and “models.” 

Overall, both CSR and non-CSR schools were engaged in activities associated with most of 
the 1 1 components, probably because both were responding to similar state and federal 
accountability requirements. However, implementation of reform at the CSR schools differed 
from implementation in the non-CSR schools on the components that were most closely 
associated with externally developed models. 

This section contains the findings from the first year of the evaluation. The findings are 
drawn from the National School-Level State Assessment Score Database , NCES Common Core 
of Data, and the school-level surveys developed for the evaluation. Only selected exhibits are in 
this section; a complete set of tabulations appears in Appendix A. 



Targeting of CSR Funds 

Funding for schools in the 2002 cohort varied. CSR schools received an average of 
$100,565 for CSR activities per year, which represents an increase of funding from $66,175 in 
1998. However, the amount varied by type of school, with high schools receiving more money 
than elementary and middle schools. Although high schools received more money, the allocation 
per student was smaller at $122 than in middle schools ($162 per student) or elementary schools 
($189 per student). Further, schools in urban districts received more funding than schools in 
suburban districts and small towns, but rural schools received more money than small town 
schools. Funding also varied by state, with California granting $387,403 as an average award and 
Maine granting $50,000. 

The CSR legislation intentionally targets CSR funds to low-performing schools that serve 
high-need students. 

CSR funds were strongly targeted to high-poverty schools and those with high 
concentrations of minority students, as well as to urban schools. Almost half (45 percent) of 
CSR schools had poverty rates of at least 75 percent, nearly three times greater than the 
percentage of all schools in this highest-poverty group (16 percent) and close to double the 
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percentage of Title I schools (26 percent) (Exhibit 2). Similarly, schools with high concentrations 
of minority students (75 percent or higher) accounted for nearly half (47 percent) of CSR 
schools, compared with 30 percent of Title I schools and 21 percent of all schools (Exhibit 3). 
CSR schools were much more likely to be located in urban areas (46 percent of CSR schools) 
than were Title I schools (26 percent) or all schools (25 percent). Rural schools were equally 
represented among CSR schools, Title I schools, and all schools (13 percent of each group). CSR 
schools were less likely to be located in suburbs and towns than in rural or urban communities 
(Exhibit 4). 

The distribution of CSR schools by poverty and minority status was similar to the 
distribution of Title I schoolwide programs — not a surprising finding, because both 
programs are targeted to high-need schools. For example, the highest-poverty schools 
accounted for 45 percent of CSR schools and 42 percent of Title I schoolwides. However, CSR 
schools were more likely to be located in urban areas (46 percent) than were Title I schoolwides 
(37 percent). The proportion of CSR schools that were operating Title I schoolwide programs 
was 56 percent, compared with 25 percent of all schools operating Title I schoolwide programs. 



Exhibit 2 

Distribution of CSR Schools and Other Schools by School Poverty Rate 




CSR Schools Title I Schoolwide All Title I Schools All Schools 
Programs 



Exhibit reads: Almost half (45 percent) of CSR schools had poverty rates of at least 75 
percent, nearly three times greater than the percentage of all schools in this high poverty 
group (16 percent) and close to double the percentage of Title I schools (26 percent). The 
distribution of CSR schools by poverty status was si mi lar to the distribution of Title I 
schoolwide programs (42 percent). 
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Exhibit 3 

Distribution of CSR Schools and Other Schools by Percentage of Minority Students 



100% 



80 % 



60 % 




40 % 



20 % 



CSR Schools Title I Schoolwide All Title I Schools 
Programs 



All Schools 



□ 75 - 100 % 

□ 50 - 74 % 
13 25 - 49 % 
■ 0 - 24 % 



Exhibit reads: Schools with high concentrations of minority students (75 percent or 
higher) accounted for nearly half of CSR schools and Title I schoolwide programs (47 
percent each), compared with 30 percent of Title I schools and 21 percent of all schools. 
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Exhibit 4 

Distribution of CSR Schools and Other Schools by Urbanicity 
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CSR Schools Title I Schoolwide All Title I Schools All Schools 
Programs 



Exhibit reads: CSR schools were much more likely to be located in urban areas (46 
percent of CSR schools) than were Title I schoolwides (37 percent), Title I schools (26 
percent), or all schools (25 percent). Rural schools were equally represented among CSR 
schools, Title I schools, and all schools (13 percent of each group). 

Perhaps more important, school survey data show that CSR schools were significantly 
more likely to report that they were identified as a low-performing school according to the 
criteria used in their state at the time of award (46 percent) than were non-CSR schools (28 
percent). When they received funding CSR schools were also more likely to have received state 
sanctions due to low performance (11 percent as compared with 3 percent). These are additional 
indicators that CSR funds are being targeted to schools in need of improvement (Exhibit 5). 
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Exhibit 5 

Schools Reported Being Identified as Low-Performing or Sanctioned Because of 
Low Performance According to State Criteria 




*Difference is statistically significant at the .01 level. 

Exhibit reads: CSR schools were significantly more likely to report that they were 
identified as a low-performing school and had received sanctions due to low 
performance. 

Perhaps the most important characteristic of CSR schools is the level of student 
achievement prior to receiving funds. Because student achievement is measured and reported 
differently across states, available scores cannot be directly compared. In order to summarize 
data across states in the simplest manner, scores within each state were first transformed into z- 
scores. Z-scores are centered on state means and are scaled in units of standard deviations. Thus, 
the average z-score for all schools, across all states is, by definition, zero. In this case, a negative 
average z-score indicates that the performance of a group is below state averages, across all 
states. Reports in subsequent years will use meta- analytic techniques to aggregate data across 
states. (For more information about this year's calculation of student achievement z-scores, see 
Appendix B.) 

To compare baseline student achievement scores across funding groups, average school- 
level z-scores were calculated in reading and mathematics for each school level. CSR schools 
had lower baseline achievement scores than did Title I schoolwides in reading and 
mathematics at most grade levels (elementary, middle and high school in math; elementary and 
middle in reading) at the time awards were made (Exhibit 6). Further, the difference was true 
regardless of school locale or poverty level (Exhibit A-l). 
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Exhibit 6 

Average School-Level Z- Scores for Math and Reading 
at CSR Schools (2002 Cohort), Title I Schools, and All Schools 



Math 


CSR 

Schools 


Title I 
Schoolwides 


All Title I 
Schools 


All Schools 


Elementary 


-0.95 


-0.50 


-0.24 


0.0 


Middle 


-0.86 


-0.46 


-0.19 


0.0 


High 


-0.68 


-0.66 


-0.24 


0.0 


Reading 


Elementary 


-0.92 


-0.54 


-0.26 


0.0 


Middle 


-0.89 


-0.49 


-0.21 


0.0 


High 


-0.69 


-0.72 


-0.25 


0.0 



Exhibit reads: CSR schools had lower baseline achievement scores in math and reading 
than did Title I schoolwides, Title I schools, or all schools. (A negative average z-score 
indicates that the performance of a group is below state averages). 

The differences between 2002 CSR schools and the population of Title I schoolwides 
points to an important policy issue. CSR schools’ lower initial scores may reflect additional 
targeting to schools with the greatest need for improvement. Interviews with state and district 
officials will explore the rationale for allocating funds. 

CSR was designed to provide support for school reform in low-performing, high-need 
schools. Funding is provided through states, which then select schools within districts for 
funding. Clearly, states are selecting schools that meet the intent of the legislation. 



Implementation of School Reform Activities in CSR Schools and Other Schools 

Both CSR and non-CSR schools reported they were implementing specific activities that 
prior research indicates are associated with reform. However, as discussed below, the CSR 
schools differed from the non-CSR schools in their implementation of components directly 
related to selecting, implementing, and evaluating models for reform. Further, CSR funding 
seems to contribute to building capacity for ongoing reform, with schools reporting more school 
activity that reflects coherence and cohesiveness during the first year of implementation (2002- 
03) compared with the previous year. 

This section begins with data related to the extent to which the 1 1 components of CSR 
were present in the CSR and comparison schools. It concludes with a discussion of the nature of 
comprehensive school reform. Because this is the first year of data collection and analysis, the 
final section is fairly speculative, pointing to areas for greater concentration in future years. (All 
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data in this section are drawn from the mailed surveys, and significance is reported at the p< .01 
level.). Because of the large association between the teacher and principal responses, only the 
principal responses are reported here as the measure of school reform. However, on a small 
number of items minor disagreements between principals and teachers existed. On those items, 
the responses for both teachers and principals are noted. These differences will be explored in 
subsequent reports. 



School Improvement Plans 

Nearly all schools in the sample (CSR and non-CSR schools) reported they had formal 
comprehensive plans for school reform (93 percent for CSR schools; 89 percent for non-CSR 
schools) (Exhibit A-2). Principals at both CSR and non-CSR schools indicated that these plans 
included components similar to the 11 CSR components, although CSR schools were more likely 
than non-CSR schools to report seeking research evidence about a proposed reform and adopting 
a reform created outside of the school. 

According to respondents, nearly all improvement plans for both CSR and non-CSR 
schools included measurable goals and objectives (98 percent in both groups) and professional 
development activities (94 percent in both groups). Both were highly likely to include curriculum 
and instruction (90 percent and 89 percent), a mechanism for periodic evaluation of goals (88 
percent and 84 percent), and a plan for parental involvement (83 percent and 81 percent). Both 
were less likely to include classroom management guidelines (48 percent and 39 percent) or 
student assessment rubrics in the plan (52 percent and 49 percent) (Exhibit 7). It may be that 
state and federal regulations for underperforming schools influence all schools. As data are 
gathered from interviews with state and district policymakers, the study will explore that 
possibility. 
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Exhibit 7 

Aspects of Reform Covered by School Improvement Plans 





CSR Schools 


Non-CSR 

Schools 


Measurable goals and objectives 


98% 


98% 


Professional development activities 


94% 


94% 


Curriculum and instruction 


90% 


89% 


Periodic evaluation of goals 


88% 


84% 


Parental involvement 


83% 


81% 


Student assessment rubrics 


52% 


49% 


Management guidelines 


48% 


39% 



Exhibit reads: Nearly all school improvement plans included measurable goals and 
objectives (98 percent for CSR schools and non-CSR schools), professional development 
activities (94 percent for both groups) and plans for curriculum and instruction (90 
percent and 89 percent). 

Few differences were found in the factors that influenced the content of the school 
improvement plan. Both CSR and non-CSR schools cited state and district content standards as 
the major influence on the plan (90 percent compared with 91 percent), and, although non-CSR 
schools selected “state or district performance standards” more frequently (91 percent) than did 
the CSR schools (87 percent), the difference was not significant. However, CSR schools were 
significantly more likely than non-CSR schools to cite the specifications of a reform design 
as influencing their school improvement plan (63 percent as compared with 34 percent) 
(Exhibit A-3). This indicates that reform models are more widespread in CSR schools than in 
non-CSR schools. 



Characteristics of Reform 

Although CSR schools and non-CSR schools were equally likely to state the reform was 
“adapted with modifications from an external source” (42 percent and 41 percent) (Exhibit A-4), 
they indicated different primary designers. Recipients of CSR funds reported they were more 
likely to adopt a reform that was created outside the school than were non-CSR schools (32 
percent compared with 6 percent) (Exhibit A-5). Further, CSR schools were influenced by design 
specifications of the reform significantly more than were non-CSR schools (63 percent and 34 
percent, as mentioned above and cited earlier in Exhibit A-3). CSR schools also were more 
likely than non-CSR schools to identify a specific reform model (85 percent vs. 49 percent) 
and indicate they were implementing only one reform (67 percent vs. 57 percent). 

However, both groups were equally likely to claim participation in reform by all grades and 
the “whole school.” Further, if they were not involved in whole-school reform, both CSR and 
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non-CSR schools focused primarily on reading and language arts, followed by mathematics. 
Relatively few respondents from either group indicated that the reform involved science, social 
studies, or the arts. These similarities will be explored further in the field-based inquiry, 
interviews with district officials and a second round of survey data collection. Later analyses will 
also focus on the extent of implementation over time. 

Faculty Role 

The presence of CSR funding is associated with differences in the faculty role in reform. 
Teachers in CSR schools were significantly more likely to have voted to adopt the reform 
than in non-CSR schools (82 percent as compared with 55 percent). Further, although the 
greatest number of CSR respondents indicated that all teachers participate in the reform (81 
percent), this was not significantly different from the response from the non-CSR schools (78 
percent). However, CSR schools were more likely to be phasing in the reform (31 percent) and 
involving additional teachers over time than were non-CSR schools (21 percent), according to 
respondents (Exhibit 8). 



Exhibit 8 

Extent of Faculty or Teacher Participation in Reform at the School 




*Dijference is statistically significant at the .01 level. 



Exhibit reads: Teachers at CSR schools were significantly more likely to have voted to 
adopt the reform effort at their school (82 percent compared with 55 percent at non-CSR 
schools). CSR schools also were more likely to be phasing in the reform (31 percent) than 
were non-CSR schools (21 percent). 
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Most model developers require a faculty vote before they work with a school, so the 
association of votes with CSR is also an association with the use of a model. In addition, the 
phasing in of reform may also be associated with working closely with model developers. As 
will be seen, developers provide on-site assistance, which gives them knowledge of the school 
and faculty and may lead to their advising phasing in reform. As the evaluation continues, it will 
explore more closely whether developers influence such strategies as phasing in reform. Further, 
the evaluation will assess the extent to which actions such as faculty votes, phasing in reform, 
and receiving on-site assistance influence implementation and outcomes. 



School Performance Goals 

Almost all respondents in both groups reported having performance goals in reading and 
language arts (98 percent and 100 percent) and mathematics (94 percent and 96 percent) 

(Exhibit A-6), and less than half of either group have annual performance goals by either grade 
level (48 percent and 44 percent) or other content area (32 percent and 29 percent) (Exhibit A-7). 
(The new requirements of NCLB and the influence of state performance goals are likely to 
change such findings in the next round of data collection). In both groups, more teachers than 
principals reported that their schools had annual performance goals by grade level and content 
area (80 percent of teachers compared with 48 percent of principals in CSR schools; 78 percent 
of teachers compared with 44 percent of principals in non-CSR schools). 

However, differences existed about the factors that influence the performance goals and 
how the goals and the reform were evaluated. Each of these is discussed in turn. 

State tests were the major determinant of school performance goals for both CSR and 
non-CSR schools (87 percent and 89 percent). Both groups were also greatly influenced by state 
content standards (83 percent and 81 percent). Few in either group are influenced by parent 
concerns (28 percent and 32 percent). However, schools participating in CSR report that the 
reform effort influences their performance goals more often than do the non-CSR schools 
(57 percent vs. 41 percent) (Exhibit 9). This difference is significant. 
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Exhibit 9 

Factors Influencing the Creation of Performance Goals at the School 




*Dijference is statistically significant at the .01 level. 



Exhibit reads: State tests and state content standards were the major determinants of 
school performance goals for both CSR and non-CSR schools. However, 57 percent of 
CSR schools reported that the reform effort being implemented influenced their 
performance goals, compared with 41 percent of non-CSR schools. 

CSR schools differed from the non-CSR schools in how they evaluated their performance 
and the evidence they used to show the progress of reform. CSR schools were significantly 
more likely to have a formal plan to evaluate their progress (91 percent compared with 82 
percent). However, the greatest influence on the questions addressed in evaluation comes from 
state requirements for both CSR and non-CSR schools. Nonetheless, CSR schools were more 
likely to attend to the requirements of reform (66 percent), than were non-CSR schools (42 
percent) (Exhibit A-8). 

Equally important, CSR and non-CSR schools use different evidence for evaluation. 
Non-CSR schools use “results from students at this school” significantly more than do CSR 
schools (82 percent for the non-CSR schools and 68 percent for the CSR schools). (Because the 
information comes from the surveys, it is not possible to tell whether the results being used are 
formal evaluations or informal assessments. The field-based study will pursue this question 
further.) In contrast, the CSR schools, used research conducted by the reform designer (47 
percent compared with 26 percent for non-CSR schools) and independent research (42 percent 
vs. 26 percent for non-CSR schools). The difference in sources of evidence seems related to the 
adoption of a reform model. It may also be an indication that CSR schools’ capacity for reform is 
increasing (Exhibit 10). 
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Exhibit 10 

Source of Evidence to Link School Reform to Student Achievement 




*Difference is statistically significant at the .01 level. 

Exhibit reads: When evaluating school reform efforts, non-CSR schools use “results from 
students at this school” (82 percent) significantly more often than CSR schools (68 
percent). CSR schools, on the other hand, were more likely to rely on research conducted 
by the reform designer (47 percent) and independent research (42 percent). 



Professional Development 

Professional development is an essential component of school reform because reform 
requires teachers to learn new practices, either for classroom application or engagement in 
different forms of school organization. The surveys showed differences in professional 
development practices in CSR and non-CSR schools, in the locus of activity, the number of days 
provided, and the content. CSR schools noted changes in the kinds of professional development 
activities from last year to this year. 

Professional development was included in the school reform plan more frequently in 
CSR schools than in non-CSR schools. In addition, a greater number of CSR schools than 
non-CSR schools provided more than 10 days for professional development and received 
on-site assistance from external supporters. Ninety percent of CSR schools included 
professional development for all teachers in their school reform plan compared with 73 percent 
of non-CSR schools. CSR schools also provided more than 10 days for professional development 
more often than did non-CSR schools (56 percent as compared with 39 percent). External 
assistance providers supported reform efforts on-site in significantly more CSR schools (85 
percent) than non-CSR schools (57 percent). Finally, formal evaluation plans in CSR schools 
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were more likely to include assessment of the utility of external assistance than such plans in 
non-CSR schools (41 percent vs. 30 percent) (Exhibit 11). 



Exhibit 11 

Status of Professional Development in School Reform 





CSR schools 


Non-CSR 

schools 


Professional development for all teachers is 
included in school improvement plan 


90% 


73 %* 


School provides 10 or more days for 
professional development 


56% 


39%* 


School receives on site support for reform 
efforts from external providers 


85% 


57%* 


School evaluation plan includes assessment of 
the utility of external assistance 


41% 


30% * 



*Difference is statistically significant at the .01 level. 

Exhibit reads: In CSR schools as compared with non-CSR schools, professional 
development more often was included in the school reform plan (90 percent vs. 73 
percent), offered for 10 days or more (56 percent vs. 39 percent), and took the form of 
on-site assistance from external sources (85 percent vs. 57 percent) 

The following professional development opportunities increased significantly in CSR 
schools from 2001-02 to 2002-03 (Exhibit A-9): 

• Reading or language arts instruction. 

• Mathematics instruction. 

• Instructional strategies for low-achieving, limited English proficient, special 
education, or migrant students. 

• Ensuring that curriculum and instruction are consistent with state and district 
content standards. 

• Ensuring that curriculum and instruction are consistent with state and district 
assessments. 

• Implementation of a school reform model. 

• Monitoring individual students’ progress toward learning goals. 
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• Interpreting reports of student achievement data. 

CSR teachers reported slightly fewer professional development opportunities than 
principals related to instructional strategies for low-achieving, limited English proficient, special 
education or migrant students (7 1 percent of teachers as compared with 60 percent for 
principals). CSR teachers also reported slightly fewer professional development opportunities 
than principals related to monitoring student progress (70 percent of teachers compared with 59 
percent of principals) and interpreting reports of student data (83 percent of teachers compared 
with 69 percent of principals). Despite these differences, the rates of change reported by CSR 
teachers for these three types of opportunities from 2001-02 to 2002-03 were similar to those 
reported by CSR principals. 

In addition, teachers and principals in CSR schools reported the availability of different 
types of professional development opportunities. CSR principals were more likely than CSR 
teachers to report that teachers coached other teachers (71 percent of principals versus 59 percent 
of teachers) and made management decisions (62 percent versus 46 percent). 

For the most part, differences in professional development opportunities are differences 
associated with the formal CSR model requirements. In one area, scheduling for common 
planning time, non-CSR schools reported significantly less opportunities than did CSR schools 
(31 percent of non-CSR schools principals responded that no common planning time is 
scheduled vs. 21 percent of CSR schools). The reason for this difference will be explored further 
in the field-based study. 



Support for School Reform 

Significant differences existed between CSR and non-CSR schools in the type of support 
they received. As might be predicted, CSR schools were far more likely to receive support 
from a model developer than non-CSR schools (31 percent vs. 6 percent). However, non-CSR 
schools were more likely to report receiving support for school reform efforts from the 
district than were CSR schools (72 percent of non-CSR schools and 34 percent of CSR 
schools) (Exhibit 12). This difference will be explored further in the telephone interviews and 
case studies. 
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Exhibit 12 

Entity Primarily Responsible for Supporting Reform Efforts at the School 




*Dijference is statistically significant at the .01 level. 



Exhibit reads: CSR schools were far more likely to identify model developers as the 
primary supporters of reform at their school (31 percent for CSR schools compared with 
6 percent for non-CSR schools). Conversely, non-CSR schools (72 percent) reported 
more district support for school reform efforts than CSR schools (34 percent). 

States and districts were more likely to provide funds for reform to non-CSR schools 
than to CSR schools. Discretionary district funds went to 58 percent of non-CSR schools 
compared with 44 percent of CSR schools. Special state grants were awarded to 53 percent of 
non-CSR schools compared with 44 percent of CSR schools (Exhibit 13). 
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Exhibit 13 

Sources of Funding That Contribute to Implementation and Operation of School Reform 



100% -i 
80% - 



60% 



40% 



20% 



0 % 





58 % 








1 53 % 









Discretionary district funds* Special state grants* 



■ CSR Schools □ Non-CSR Schools 



*Difference is statistically significant at the .01 level. 

Exhibit reads: Schools reported that special state grants and district discretionary funds 
were more commonly used for reform at non-CSR schools (58 percent for district funds; 
53 percent for state funds) than at CSR schools (44 percent for each source of funds). 



Further, districts supported different kinds of activities in CSR schools than non-CSR 
schools in the first year of reform implementation. Districts were more likely to help CSR 
schools select a school reform model (45 percent in CSR schools compared with 32 percent in 
non-CSR schools) but were less likely to provide CSR schools with professional development 
for school reform (72 percent in CSR schools and 86 percent in non-CSR schools) 

(Exhibit A- 10). States also were more likely to help CSR schools select a school reform model 
(25 percent) than they were to help non-CSR schools select a model (16 percent) (Exhibit A-ll). 

Both CSR and non-CSR schools reported coordinating funds from a variety of sources to 
support professional development and align Title I activities, but neither aligned other funds 
(e.g., bilingual education) or reallocated staff positions. 3 



Instructional Practice 

CSR and non-CSR school respondents indicated little difference in instructional 
practice. Both groups reported that curriculum scope and sequence was determined at the district 



3 Although 22 percent of the non-CSR schools reported receiving federal CSR funds, later follow-up indicated that only five 
schools were actually recipients of CSR funds. Those schools have been removed from the analysis. 
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level (78 percent CSR; 83 percent non-CSR). Other loci for determining scope and sequence 
varied, but not significantly. For example more non-CSR schools reported that the state 
organized curriculum (52 percent) than did CSR schools (46 percent), and more CSR schools 
reported individual teachers (34 percent) or schools (5 1 percent) organized curriculum than did 
non-CSR schools (27 percent; 47 percent) (Exhibit 14). 



Exhibit 14 

Primary Organizer of the Scope and Sequence of Curriculum at the School 



100% 



80% 



60% 



40% 



20 % 



0% 




District 



School State 

■ CSR Schools □ Non-CSR Schools 



Individual teacher 



Exhibit reads: The district was primarily responsible for determining curriculum scope 
and sequence at CSR schools (78 percent) and non-CSR schools (83 percent). More 
non-CSR schools reported that the state organized curriculum while CSR schools were 
slightly more likely to cite individual teachers or schools as controlling curriculum and 
instruction. 

Further, about the same percentages of CSR and non-CSR schools reported participating in 
grade-level teams (86 percent of CSR respondents vs. 84 percent of non-CSR school 
respondents) or in content area teams across grades (71 percent, CSR vs. 75 percent non-CSR). 

As implementation continues, changes may be observed with regard to scope and sequence, 
but little change in responses related to teaming is expected, given the high percentages that 
currently exist. Additional information will come from the field-based studies. 
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The Role of Parents 



CSR and non-CSR schools did not differ in how parents were involved in reform. 

Both communicated with parents and encouraged parent involvement in similar ways. They were 
also similar in their assessment of the ways parents were engaged with the school. 

Both groups communicated with parents most frequently through telephone calls (95 
percent for CSR schools; 94 percent for non-CSR schools), regular newsletters (83 percent CSR 
and 86 percent non-CSR) and in the language other than English spoken at home (45 percent, 
CSR; 46 percent, non-CSR). However more CSR schools reported having a parent coordinator 
(43 percent) than did non-CSR schools (36 percent) although the differences were not 
significant. (Exhibit 15). 



Exhibit 15 

Methods of Formal Communication from Schools to Parents 




Telephone calls Regular newsletters Language spoken at Designated parent 

home (not English) coordinator 



■ CSR Schools □ Non-CSR Schools 



Exhibit reads: CSR and non-CSR schools communicated with parents most frequently 
through telephone calls and newsletters. Slightly more CSR schools reported having a 
parent coordinator (43 percent) than non-CSR schools (36 percent). 

The schools encouraged parental involvement in governance with regard to fundraising (79 
percent for CSR schools vs. 86 percent for non-CSR schools), defining the school mission and 
goals (75 percent CSR schools vs. 77 percent, non-CSR schools) and evaluating school 
performance (67 percent CSR schools vs. 68 percent non-CSR schools) (Exhibit 16). Parents 
were not involved in choosing instructional materials, hiring teachers and staff, or developing the 
calendar in either type of school. 
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Exhibit 16 

Ways Parents Are Engaged and Involved at School 



100% 



80 % 



60 % 



40 % 



20 % 



0% 




Fundraising 



Defining school mission and Evaluating school peformance 
goals 



■ CSR Schools □ Non-CSR Schools 



Exhibit reads: Most commonly, schools reported parent involvement in fundraising 
activities. Three quarters of the schools solicited parent input when defining the school 
mission. Fewer schools reported parent involvement in evaluating school performance. 



Types of School Reform Models or Strategies Being Implemented 

The distribution of reform models in the CSR and non-CSR samples was determined by 
examining the responses to the survey item, “Does the primary reform effort at your school have 
a name? If so, write the name here.” The higher number of named reforms in CSR sample 
schools suggests that CSR schools are adopting externally developed models more than 
non-CSR schools (Exhibit 17). Of the 85 percent of CSR schools that responded their primary 
reform was named, 80 percent of CSR principals named models that were focused on the entire 
school. In contrast, of the 49 percent of non-CSR schools that had a named reform, 78 percent of 
principals indicated a whole-school model. The remainder in each group was subject specific, 
such as literacy reforms. 
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Exhibit 17 

Schools, by Type, with Primary Reform Efforts Identified by Name 





CSR Schools 


Non-CSR Schools 


Elementary 


87% 


52% 


Middle 


81% 


45% 


High 


87% 


42% 


Weighted Average 


85% 


49% 



Exhibit reads: On average, 85 percent of CSR schools identified their primary reform 
effort with a specific name, suggesting the presence of an externally developed model, 
compared with half of the non-CSR schools. CSR schools at each grade level were 
equally likely to identify their reform effort with a specific name. 



CSR schools implemented a greater diversity of reforms than did non-CSR schools. 

CSR schools named 38 different reforms and non-CSR schools 28 (excluding generic names 
such as “school improvement program ’). In CSR schools, the most frequently named models 
were the products of private developers. However, in non-CSR schools, other than Success for 
All, regional school accreditation and state programs were the most frequently mentioned 
(Exhibit 18). 



Exhibit 18 

Number and Type of Named Primary Reform Efforts in Use at Schools 





CSR Schools 


Non-CSR Schools 


Number of 
Whole School 
Reforms Named 


38 


28 




Success for All 




Most Frequently 
Named Models 


Renaissance 

Co-nect 

America’ s Choice 
Direct Instruction 


Success for All 
State 
Regional 



Exhibit reads: CSR schools identified a more diverse list of reforms in use at 
their schools. Further, the most frequently named models in CSR schools were 
the products of private developers, as compared with non-CSR schools. 



32 






CSR and Developing Capacity for Reform 

Data that focus on differences between the first year of CSR (2002-03) and the prior year 
(2001-02) assess whether CSR increases schools’ capacity for reform. As noted earlier, CSR 
schools in the sample showed lower student achievement and higher poverty than non-CSR 
schools. They were also more likely to have been identified or sanctioned as low performing 
according to the criteria used in their state. Consequently, although changes in practice related to 
capacity for reform are limited, they are important because CSR schools must show greater 
improvement than non-CSR schools to provide students with opportunity to learn high standards. 

Newmann et al. (2001) state that the three following conditions are important in schools' 
capacity to ensure the coherence and cohesiveness of reform. 

• A common framework to guide curriculum, instruction, assessment, and the 
learning climate. 

• Staff working conditions to support implementation of the framework, 
including professional development for staff. 

• School allocation of resources to support implementation. 

Most CSR schools are working toward a common framework. The framework is shaped by 
the model adopted as well as state and district performance standards. The existence of a 
framework in 2002-03 represents a major change from the previous year. A higher percentage 
of CSR principals reported their schools had a comprehensive written plan in 2002-03 (93 
percent) as compared with the previous year (75 percent), indicating some influence of CSR. 
In contrast, principals in non-CSR schools reported little change (89 percent had comprehensive 
written plans in 2002-03, compared with 86 percent in 2001-02). In addition, CSR schools in the 
sample were significantly more likely to report engaging in whole school reform in 2002-03 
(76 percent) than in 2001-02 (55 percent). As indicated above, they were more likely to report 
implementing a single reform effort, increasing coherence. 

Although CSR schools reported little difference in how the instructional staff was 
organized and fewer cross-subject area or within-grade teams than the non-CSR schools, 
teachers in CSR schools did report more opportunities for professional development. Teacher 
participation in grade-level or content area teams increased significantly in 2002-03, compared 
with the previous year (Exhibit 19). In addition, CSR teachers reported receiving more days of 
professional development in 2002-03 than the year prior, and the training was more focused on 
issues related to reform (Exhibit 20). 
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Exhibit 19 

Teacher Enhancement Opportunities Available 
before and after Reform Implementation 




*Difference is statistically significant at the .01 level. 

Exhibit reads: Compared with pre-implementation levels, teachers in CSR schools 
reported more opportunities to work on grade-level teams (86 percent vs. 73 percent the 
previous year) and content area teams (71 percent vs. 58 percent) during the first year of 
CSR. 
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Exhibit 20 



Types of Professional Development in Which Teachers Participated in during the First 
Year of Reform Implementation (2002-03) Compared with the Previous Year 




*Dijference is statistically significant at the .01 level for all items. 



Exhibit reads: CSR teachers were much more likely to receive training on model 
implementation (85 percent), monitoring students’ progress (70 percent), and interpreting 
reports (83 percent) in 2002-03, compared with 2001-02 (the year prior to CSR 
implementation). 

On the other hand, the CSR schools were not coordinating resources to any greater 
extent than non-CSR schools in support of reform efforts at the school (28 percent 
compared with 35 percent). CSR schools were using the federal CSR funds to support the 
reform, which indicates that capacity to continue reform may end with the end of the CSR grant. 
The importance of coordinated resources will be assessed as the evaluation continues 

At the end of a year of implementation, CSR has increased participating schools’ capacity 
to reform to some extent on two key dimensions, developing comprehensive written plans and 
increasing team meetings. It has yet to produce changes in how schools coordinate resources. 

The Nature of Comprehensive School Reform 

This report addresses the question: "What does comprehensive school reform look like at 
the end of the first year of implementation?" This section summarizes findings in response to that 
question. In addition, CSR is intended to increase schools’ capacities to reform. This section also 
summarizes findings relevant to reform capacity. 
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Summary of CSR Implementation 



CSR comprises 11 components whose existence and interaction may improve schools. 
Respondents to the survey indicated that both CSR and non-CSR schools were implementing a 
number of the components. However, CSR schools were more likely than non-CSR schools to 
implement components most associated with adopting a model. Consequently, the presence of 
some similar components in CSR and non-CSR schools may not indicate equal progress toward 
reform nor lead to equal outcomes for students. The differences in the components that are 
implemented in CSR and non-CSR schools may well encompass different interactions, which, in 
turn affect the extent to which schools are coherent and cohesive, enabling them to provide 
students with focused and challenging opportunities to learn to high standards. 

CSR schools, as compared with non-CSR school, were more likely to implement the 
following components: 

• Adopt externally developed methods and strategies that have been replicated. 

They did so by: 

Identifying a specific reform model (85 percent compared with 49 
percent). 

Using evidence from research that the reform model chosen improves 
student achievement (42 percent compared with 26 percent). 

• Provide more continuous professional development. They did so by: 

Including professional development activities for all teachers (90 percent 
compared with 73 percent). 

Allocating over 10 days to teacher professional development (56 percent 
compared with 39 percent). 

• Include measurable goals for student performance associated with the reform 
model (57 percent compared with 41 percent). 

• Reflect support from staff by including a formal vote by teachers for the 
reform model (82 percent compared with 55 percent). 



• Provide support for staff by receiving on-site consulting relevant to the reform 
(85 percent compared with 57 percent). 

• Evaluate the reform. They did so by: 

Including the requirements of the reform model in the scope and content 
of evaluation (66 percent compared with 42 percent). 
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Assessing the utility of external assistance (41 percent compared with 30 
percent). 



To some extent, the differences are associated with the adoption of models by the CSR- 
funded schools. At this point of early implementation, the evaluation cannot assess whether some 
models lead to greater implementation and outcomes than others. However, the act of selecting a 
model has some immediate outcomes. For example, most models require faculty votes as 
indications of buy-in. The vote itself may create the conditions for coherent implementation, and 
when accompanied by on-site assistance (provided by the model developer), strengthen the 
probability of full implementation. Further, models frequently include performance goals, so the 
CSR schools are more likely to have reform- specific performance goals. Model developers also 
require professional development, and even when they do not require it, offer model-focused 
opportunities, so the higher number of days for professional development can also be associated 
with adoption of a model. 

In sum, both CSR and non-CSR schools exhibited many aspects of comprehensive 
reform. However, CSR schools were more likely to adopt externally developed models. 
Other differences between the two types of schools were related to model adoption. 



The Influence of State and District Policies on the Implementation of 
Comprehensive Reform Programs 

State and local policies influence CSR and non-CSR schools in similar ways, except 
with regard to selecting the reform model. Reform plans in both types of schools are highly 
influenced by state or district content and performance standards (90 percent of CSR schools and 
91 percent of non-CSR schools on content standards; 87 percent of CSR schools and 91 percent 
of non-CSR schools on performance standards) (Exhibit 21). Similarly, state content standards 
and testing requirements have the greatest influence on performance goals at both CSR and 
non-CSR schools. Further, districts, rather than states or individual teachers, have the most 
influence on curriculum scope and sequence in both CSR and non-CSR schools (5 1 percent in 
CSR schools and 47 percent in non-CSR schools). 
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Exhibit 21 

Influence of State and Local Policies on School Improvement Plans 



100% 



80% 



60% 



40% 



20% 



0% 




Exhibit reads: Most principals in CSR and non-CSR schools report that content and 
performance standards influence school improvement plans. 

Both groups of schools report control over budgets and personnel decisions similarly (82 
percent for CSR and 75 percent for non-CSR with budgetary control; 78 percent for CSR and 80 
percent for non-CSR regarding personnel decisions) (Exhibit 22). 
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Exhibit 22 

Control of Budget and Personnel in CSR and Non-CSR Schools 



100% 



80% 



60% 



40% 



20% 



0% 




Exhibit reads: The majority of CSR and non-CSR schools report that budgetary and 
personnel decisions are controlled at the school level. 

Teachers and school administrators were involved in selecting the reform model or 
approach being implemented, both at CSR schools and non-CSR schools. However, the 
school board and the district central office played a more significant role in selecting 
reform at non-CSR schools than at CSR schools, indicating more “top-down” requirements 
for changes in practice. One third of CSR schools reported that the district central office was 
one of several entities responsible for selecting the reform, compared with 57 percent of 
non-CSR schools. Non-CSR schools also reported that school board members were involved in 
the decision at a higher rate than did CSR schools (24 percent for non-CSR schools compared 
with 15 percent of CSR schools) (Exhibit 23). Further, state or district mandates were more 
likely to contribute to the selection of reform at non-CSR schools (60 percent) than at CSR 
schools (31 percent). 
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Exhibit 23 

Entities Involved in Selecting a Reform Model or Approach at the School 



100% 



80% 



60% 



40% 



20 % 



0% 



76% 



63% 




School Teachers School District central Parents School Board* 

improvement administrators office* 

team 



■ CSR Schools □ Non-CSR Schools 



* Difference is statistically significant at the .01 level. 



Exhibit reads: School staff were involved in selecting a reform model or approach at both 
CSR and non-CSR schools. However, the district central office (57 percent) and school 
boards (24 percent) had a greater role at non-CSR schools than at CSR schools (34 
percent; 15 percent). 
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IV. Summary 



This final chapter summarizes findings related to school reform implementation. The 
following section presents policy implications. 



Summary of Findings 

CSR funds have been targeted to high-need schools as intended. Participating schools tend 
to have high poverty rates and high concentrations of minority students, and have frequently 
been identified as low-performing according to state criteria. 

Both CSR and non-CSR school personnel view themselves as engaged in activities 
associated with school reform. Further, activities include the majority of the 11 components 
identified in NCLB. However, CSR schools are more intensively implementing components 
most associated with adopting a research-based model. They were more likely to: 



• Adopt externally developed methods and strategies that have been replicated. 

• Provide more continuous professional development. 

• Include measurable goals for student performance associated with the reform 
model. 

• Reflect support from staff by including a formal vote by teachers for the 
reform model. 

• Provide support for staff by receiving on-site consulting relevant to the 
reform. 

• Evaluate the reform. 

Consequently, the Comprehensive School Reform program is, at the early stage of 
implementation in the schools in the sample, affecting how they address low performance. While 
CSR and non-CSR schools are implementing many components of reform, CSR influences how 
schools approach reform. CSR schools are likely to adopt a single, externally developed reform 
model. However, they are not more likely to integrate budgets than are other schools, which may 
have longer-term consequences for the reform effort. The evaluation will seek to determine 
whether the differences between CSR and non-CSR school reform affects implementation and 
outcomes over time. 
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Implications 



The first year of the evaluation has yielded information with implications for federal 
policy. The implications relate to two key findings: 

• Although both CSR and non-CSR schools are engaged in reform, reform in 
CSR schools includes adoption of models and other activities closely 
associated with research-based models. 

• CSR funds are strongly targeted to high-poverty schools and low-performing 
schools, and schools receiving CSR funds are lower performing than are other 
schools with similar demographic characteristics. 

Taken together, the findings raise interesting questions. The first is whether the use of CSR 
funds accelerates reform in the lowest performing schools. States and districts seem to have 
targeted CSR funds to those schools that have the greatest need to change practices in order to 
support high achievement for all students. With CSR funds, the schools were more likely to 
adopt models, focus professional development on reform, and track student performance than 
were non-CSR schools. Both CSR and non-CSR schools were engaged in other reform activities. 
In subsequent years, the evaluation will provide information about whether CSR schools 
implement more reform components more thoroughly than do the non-CSR schools. If they do, 
CSR can be seen as adding value to improvement by providing a mechanism that focuses efforts 
and enables school staff members to organize themselves in ways that offer greater educational 
opportunities for students. Perhaps CSR helps schools jump-start improvement. 

Second, data from the first year of this evaluation indicate that all schools in the sample are 
engaged in many aspects of what the legislation defines as “comprehensive school reform.” 
Consequently, the study has implications about the nature of reform in general. Most low- 
performing schools in the non-CSR group are making efforts to improve. Questions then arise as 
to whether the efforts are associated with improved outcomes: Do schools succeed in reform 
without models to organize them? Are models only important in the lowest performing schools? 
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Appendix A: Selected Data Tabulations 



Exhibit A-l 

Baseline Student Achievement Scores 



Averaged Baseline Student Achievement Z-Scores** 



MATH 


Elem. 


Middle 


High 


READING 


Elem. 


Middle 


High 


2002 CSR 








2002 CSR 








Cohort 


-0.95 


-0.86 


-0.68 


Cohort 


-0.92 


-0.89 


-0.69 


SW Title I 


-0.50 


-0.46 


-0.66 


SW Title I 


-0.54 


-0.49 


-0.72 


Title I 


-0.24 


-0.19 


-0.24 


Title I 


-0.26 


-0.21 


-0.25 


All schools 


0.00 


0.00 


0.00 


All schools 


0.00 


0.00 


0.00 


Distribution of 2002 CSR 


Cohort Average Z-Scores by Poverty Level 








MATH 


Elem. 


Middle 


High 


READING 


Elem. 


Middle 


High 


75-100% 


-1.30 


-1.22 


-1.10 


75-100% 


-1.30 


-1.23 


-1.05 


50-74 


-0.59 


-0.60 


-0.88 


50-74 


-0.54 


-0.71 


-0.99 


25-49 


-0.27 


-0.33 


-0.51 


25-49 


-0.22 


-0.29 


-0.41 


0-24 


-0.62 


-0.32 


-0.25 


0-24 


-0.67 


-0.07 


-0.38 



Distribution of 2002 CSR 


Cohort Average Z-Scores by Locale 








MATH 


Elem. 


Middle 


High 


READING 


Elem. 


Middle 


High 


Urban (1&2) 


-1.31 


-1.22 


-1.09 


Urban (1&2) 


-1.29 


-1.22 


-1.11 


Suburb. (3&4) 


-0.83 


-0.84 


-0.49 


Suburb. (3&4) 


-0.84 


-0.88 


-0.57 


Town (5&6) 


-0.37 


-0.36 


-0.81 


Town (5&6) 


-0.38 


-0.36 


-0.58 


Rural (7&8) 


-0.28 


-0.27 


-0.25 


Rural (7&8) 


-0.34 


-0.32 


-0.24 



**Data from some states were not available from the 2002 school year. See achievement indicators table for details. 



Exhibit reads: CSR schools had lower achievement scores in math and reading than did 
Title I schoolwides, Title I schools, or all schools. Within the 2002 CSR cohort, schools 
with high poverty rates and urban locales scored lowest on reading and math achievement 
tests 
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Exhibit A-2 

CSR and Non-CSR Schools Reporting Formal Comprehensive School Improvement Plans 



100% 



80% 



60% 



40% 



20% 



0% 




2002-03 



2001-02 



I CSR Schools □ Non-CSR Schools 



Exhibit reads: A higher percentage of CSR schools reported having formal school 
improvement plans in 2002-03 (the first year of CSR implementation) as compared with 
the year prior to CSR implementation (93 percent compared with 75 percent). The 
number of non-CSR schools reporting formal school improvement plans over that time 
increased marginally (89 percent compared with 86 percent). 
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Exhibit A-3 

Factors Influencing the Content of the School Improvement Plan 



100% 

80% 

60% 

40% 

20% 

0% 



91% 




State or district 
content 
standards 



State or district Needs identified 
performance through a school 
standards needs 

assessment 



School 

performance 

standards 



Specifications of Assignment by 
adopted/adapted district/state 
reform design* 



■ CSR Schools DNon-CSR Schools 



*Difference is statistically significant at the .01 level. 

Exhibit reads: CSR and non-CSR schools were equally likely to cite state and district 
content standards as the major influence on their school improvement plan (90 percent; 
91 percent). CSR schools were significantly more likely to cite the specifications of a 
reform design as influencing their school improvement plan (63 percent as compared 
with 34 percent). 
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Exhibit A-4 

Source of Reform Plans at CSR and Non-CSR Schools 




Adapted with Totally designed at the Adapted selected parts Adapted unmodified 
modifications from school from external source from external source 

external source 



■ CSR Schools □ Non-CSR Schools 



Exhibit reads: CSR schools and non-CSR schools were equally likely to report that their 
reform efforts were “adapted with modifications from an external source” (42 percent and 
41 percent). Non-CSR schools were slightly more likely than CSR schools to be 
implementing a reform “totally designed at the school.” 
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Exhibit A-5 

The Primary Designer for the Reform at Your School 




*Dijference is statistically significant at the .01 level for all items. 

Exhibit reads: Schools receiving CSR funds were more likely that non-CSR schools to 
adopt a reform designed by a private developer or publisher (32 percent compared with 6 
percent). Nearly 90 percent of non-CSR schools reported implementing reforms that were 
designed locally (51 percent) or by the district (37 percent), compared with just over half 
of CSR schools reporting the same. 
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Exhibit A-6 




Exhibit reads: Almost all CSR and Non-CSR schools report performance goals in reading 
or language aits and mathematics. 
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Exhibit A-7 

Types of School Performance Goals for Students at CSR and Non-CSR Schools 



100% 



80% 



60% 



40% 




20% 



By grade level 



By content area 



■ CSR Schools □ Non-CSR Schools 



Exhibit reads: Less than half of CSR or non-CSR schools have annual performance goals 
by either grade level or content area. 
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Exhibit A-8 

Factors Influencing the Evaluation of School Performance Goals 




*Difference is statistically significant at the .01 level. 

Exhibit reads: A significant difference existed in the consideration of the requirements of 
reform, with 66 percent of CSR schools reporting that they include such requirements in 
their evaluation as compared with 42 percent of non-CSR schools. 
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Exhibit A-9 

Professional Development Opportunities Available to Teachers in 
the First Year of CSR Reform (2002-03) Compared with the Previous Year 





2001-02 


2002-03 


Reading/language arts instruction 


84% 


94% 


Mathematics instruction 


73% 


83% 


Instructional strategies for low-achieving, limited English 
proficient, special education, and/or migrant students 


54% 


71% 


Ensuring that curriculum and instruction are consistent with 
state and/or district content standards 


67% 


79% 


Ensuring that curriculum and instruction are consistent with 
state and/or district assessments 


61% 


69% 


Implementation of a school reform model 


46% 


85% 


Monitoring individual students’ progress toward learning 
goals 


50% 


70% 


Interpreting reports of student achievement data 


63% 


83% 


Exhibit reads: Professional development opportunities increased significantly at CSR 
schools during the first year of CSR implementation. Professional development around 
reading or language arts and mathematics instruction was most frequent. 


Exhibit A-10 






Type of Support Available through the District for Reform Efforts 




CSR 


non-CSR 


Administering a needs assessment 


45% 


47% 


Providing additional school staff to support school reform 


41% 


43% 


Selecting a school reform model 


45% 


32% * 


Writing grants to support school reform 


64% 


56% 


Providing professional development for school reform 


72% 


86%* 


Release time for teachers 


57% 


66% 


None, the district does not supply additional support 


6% 


2%* 



*Difference is statistically significant at the .01 level. 

Exhibit reads: School districts were more likely to assist CSR schools in selecting school 
reform models than non-CSR schools (45 percent compared with 32 percent). In 
non-CSR schools, districts were more likely to provide professional development for 
school reform (86 percent vs. 72 percent in CSR schools). 
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Exhibit A-ll 

Type of Support Available through the State for Reform Efforts 





CSR 


non-CSR 


Administering a needs assessment 


22% 


21% 


Selecting a school reform model 


25% 


16%* 


Writing grants to support school reform 


27% 


35% 


Providing professional development for school reform 


51% 


49% 


Release time for teachers 


13% 


18% 


None, the state does not supply additional support 


26% 


27% 



*Dijference is statistically significant at the .01 level. 



Exhibit reads: Support from states was significantly higher for selecting a reform model 
in CSR schools (25 percent) vs. non-CSR schools (16 percent) otherwise the level of state 
support for CSR and non-CSR schools is not significantly different. Note, nearly half the 
schools reported that states also provided professional development to support reform in 
both CSR and non-CSR schools. 
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Appendix B: Technical Appendix 



Sample and Comparison School Selection for LACIO 

The evaluation team sampled Comprehensive School Reform (CSR) schools in two stages. 
In the first stage, researchers selected a large number of schools randomly from the universe of 
schools that received CSR funding in 2002 and selected matched comparison schools for each 
sample school. Schools for the field-based study were selected in the second stage. From a 
random sub-sample of schools researchers deliberately selected schools to ensure that the sample 
was distributed evenly geographically and by school level. 



Selection of School Survey Sample 

The evaluation team ensured that the diversity in the population of schools was maintained 
in the sample by randomly selecting a large enough number of CSR schools to complete surveys. 
The sample of 400 schools is about 36 percent of the universe of approximately 1,100 schools 
reported to receive CSR funds for calendar year 2002. 

Researchers obtained the most complete list of CSR schools from the Southwest 
Educational Development Laboratory (SEDL). SEDL maintains this database under contract to 
the U.S. Department of Education. The Department of Education requires states to report their 
awardees to SEDL; however, the database did not include all states in the year 2002 for several 
reasons. Although SEDL continuously collects data, at any given time the data may be 
incomplete due to delinquent reporting by state education agencies. In addition, many states only 
award CSR funds to schools biennially or irregularly. As a result, the universe for calendar year 
2002 comprised 38 states and 1,096 schools out of an estimated potential number of 2,000 
awards. 

Using random sampling only, researchers chose schools that were representative of the 
population without stratification for the survey sample. The study does not meet the requirements 
for stratification because there are no subpopulations (schools) that are either domains of the 
study or require different study procedures (Kish 1995). However, in order to represent the 
diversity of policy environments where CSR is being applied, researchers took some school 
characteristics into account to check the representativeness of the sample. As shown later in this 
section, the distributions of the study sample across locale and school level were comparable to 
the distributions of the CSR universe. 

During the preparation of this report, the Department of Education allocated additional 
funds to study 15 states that were not included in the original random sample. Of the initial 38 
states in the 2002 CSR universe, three states were left out as a by-product of the random 
selection process. Five additional states reported data to SEDL after the initial sample selection. 
The remaining seven states did not make any awards in 2002 but have begun to report 2003 data 
to SEDL. Researchers selected an additional 100 schools from these missing states for two 
reasons. First, in order to be representative of all states, the sample should include schools from 
all states. Also, in order to measure the value added by CSR over Title I schoolwides, a larger 
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comparison group of Title I schoolwides was required. Thus in the second sample, the choice of 
comparison schools was limited to those classified as Title I schoolwides in 2002. Survey data 
from this sample will be included in the next annual report. 



Selection of Comparison Schools 

Many studies show that school-level student performance is influenced by school 
background characteristics such as student socioeconomic status and ethnicity. In fact, within 
each state, by regressing only the percentage of students receiving free and reduced-price lunches 
and the percentage of nonwhite students on a composite of reading and mathematics 
performance, 15-80 percent of the variation in student performance can be explained. In order to 
make a reasonable comparison between the CSR schools and other schools, researchers selected 
comparison schools to have matching background characteristics. 

Researchers used a two-step process to select potential matches for comparison with CSR 
schools in this study. First, the team created a school equivalency index for all schools in each 
state (where data were available) and then calculated a proximity score between each pair of 
schools within a state. Matching schools were selected that had the closest proximity on the 
index to CSR schools within the same district. 

A regression-based approach to weighting and combining background characteristics was 
used to construct the index of school similarity for each state. This method is a simplified version 
of the California School Characteristics Index (Technical Design Group of the Advisory 
Committee for the Public Schools Accountability Act of 1999). Using the National School-Level 
State Assessment Score Database developed by the American Institutes of Research, the team 
regressed measures of student academic performance on measures of schoolwide participation in 
federally subsidized free or reduced-price lunch programs and on schoolwide counts of student 
ethnicity. The estimated coefficients led to a composite of background characteristics for each 
school. In short, each background characteristic was weighted by the amount that it contributed 
to student performance. 

The variety of reporting formats used by states necessitated construction of a separate 
index for each state. For example, some states reported percentile rank while others reported 
percent above cut-points (usually quartiles or proficiency standards). The index reflected 
whatever measure was available, limiting comparisons among schools to within states. States 
also varied in the grades and subjects tested. The constructed indices for elementary, middle, and 
high school levels used scores from the third, seventh, and eleventh grade preferentially. If 
scores for these grades were not available, the closest available grade was used. The most 
consistently reported subject scores across states were results of reading assessments. With few 
exceptions, the reading scores were used to construct indices. The most current set of scores was 
from the 1999-2000 or 2000-01 school years in most states. Exhibit B-l summarizes the 
performance indicators used for each state, as well as the strength of the association between the 
predictor variables and performance. 
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Exhibit B-l 



Summary of Performance Indicators by State 



State 


Year 

tested 


Subject 


Type of 
measure* 


Elementary 

R 2 


Middle 

R 2 


High 

R 2 


Arizona 


2000 


Reading 


PR 


0.54 


0.55 


0.42 (10th) 


Arkansas 


2000 


Total 


PR 


0.49 (5th) 


- 


- 


California 


2001 


Reading 


PR 


0.76 


0.53 


0.37 


Colorado 


2000 


Reading 


CT 


0.72 


0.82 


- 


Delaware 


2000 


Reading 


PR 


- 


- 


- 


Florida 


2001 


Reading 


CT 


0.67 (4th) 


0.36 (8th) 


0.24 (10th) 


Georgia 


2000 


Reading 


PR 


0.56 


0.41 (8th) 


- 


Hawaii 


1999 


Reading 


CT 


0.44 


- 


- 


Iowa 


- 


- 


- 


- 


- 


- 


Idaho 


2001 


Reading 


PR 


- 


0.21 


0.18 (10th) 


Indiana 


2000 


Reading 


PR 


0.47 


- 


- 


Kentucky 


- 


- 


- 


- 


- 


- 


Louisiana 


2000 


Total 


PR 


0.63 


0.56 


0.55 (9th) 


Massachusetts 


2001 


Reading 


CT 


0.10 (4th) 


- 


- 


Maryland 


2000 


Reading 


CT 


0.50 


0.49 (8th) 


- 


Maine 


- 


- 


- 


- 


- 


- 


Michigan 


2000 


Reading 


CT 


0.21 (4th) 


0.20 


- 


Minnesota 


2000 


Reading 


CT 


0.28 


- 


- 


Mississippi 


2000 


Reading 


PR 


0.56 


0.58 


- 


Missouri 


2000 


Language 


CT 


0.34 


0.34 


- 


North Carolina 


2000 


Reading 


PR 


0.48 


0.48 


- 


North Dakota 


- 












New Hampshire 


2000 


Reading 


CT 


0.09 


- 


- 


Nevada 


1998 


Reading 


CT 


0.48 (4th) 


- 


- 


New York 


2001 


Reading 


CT 


0.67 (4th) 


0.52 (8th) 


- 


Ohio 


2000 


Reading 


CT 


0.61 (4th) 


0.62 (6th) 


- 


Oregon 


- 


- 


- 


- 


- 


- 


Pennsylvania 


1999 


Reading 


CT 


0.69 (5th) 


- 


- 


South Carolina 


2000 


Reading 


CT 


0.57 


- 


- 


Tennessee 


- 


- 


- 


- 


- 


- 


Utah 


2000 


Reading 


PR 


0.54 


0.48 (8th) 


0.50 


Virginia 


2001 


Reading 


PR 


0.58 (4th) 


- 


0.45 (9th) 


Washington 


2001 


Reading 


PR 


- 


- 


- 


Wisconsin 


2000 


Reading 


PR 


0.46 (4th) 


0.42 (8th) 


- 



*PR — percentile rank, CT — cut 

** R~ — for 3rd, 7th and 11th grade scores regressed on percent of students receiving free or reduced-price lunches 



and minority status unless otherwise noted 



After eliminating schools that previously participated in CSR, the team selected potential 
matches for each first-year CSR school based upon a minimum distance criterion. This method 
was used in the majority of cases. However alternative methods were used in two circumstances 
First, in some districts (or states) either demographic or performance data were not available. 
Second, a suitable comparison school was not available within the same district because the 
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district was too small or all other comparable schools had previously participated in CSR. Each 
of these contingencies is outlined below. 



In some cases, not enough data were available to construct a school equivalency index. 
Where states or districts do not report participation in the free or reduced-price lunch program or 
ethnicity, schools were ranked within districts using only achievement scores. Examples of states 
where these data are not available are Tennessee and Washington. Achievement data were 
missing for some schools or districts. This was often the case in high schools where the SAT 
takes the place of district-administered standardized tests. The proximity scores in these cases 
were based on an unweighted composite of the number of students qualifying for free or 
reduced-price lunches and ethnicity. Finally, in cases where neither achievement nor 
demographic data were available, comparison schools were matched by school grade span, size, 
and locale. 

In districts where a comparison school could not be selected, the team searched for a 
suitable comparison in an adjacent district of similar locale. Because the school equivalency 
index included all public schools in the state, the proximity of any school within the state could 
be calculated. The same criteria were used for selection across districts where data were 
available. In cases where data were not available, the team used the same procedures that applied 
to selecting comparison schools within districts. Of the 400 sample schools, 319 comparison 
schools were within the same district and 81 were from a different district. 



Composition of the CSR Sample and Non-CSR Sample 

LACIO focuses on questions of implementation and outcomes, linking school reform 
activities in diverse settings to student achievement. To achieve the goals of the study, 
researchers compared achievement in the universe of CSR schools with a random sample of 
schools receiving CSR funding and a comparable group of non-CSR schools. The baseline 
establishes the representativeness of the sample and the validity of the comparison group. In 
brief, findings indicate the sample was similar to all CSR schools and the comparison schools on 
school level and locale. And, although the universe and sample were similar in level of poverty, 
the comparison schools were less likely to serve the highest poverty students. The universe of 
CSR schools had lower initial achievement than the sample schools, which had lower 
achievement than the comparison schools. This section presents the baseline data and then offers 
some possible explanations for the differences. 

Both CSR and comparison schools included in the sample were similar to the universe of 
CSR schools on level of school and locale. For example, 62 percent of the universe, 67 percent 
of the sample, and 67 percent of comparison schools are elementary schools (Exhibit B-2). 
Further, 46 percent of the universe, 46 percent of the sample, and 45 percent of the comparison 
schools are urban, with similar comparability for rural schools (19 percent, 22 percent, and 22 
percent) (Exhibit B-3). 



56 




Exhibit B-2 

Type of School within All 2002 CSR Schools, CSR Sample, 
and Non-CSR Comparison Sample 



CSR Universe 


CSR Sample 


Comparison 


Elementary 


62% 


67% 


67% 


Middle 


20% 


15% 


15% 


High 


14% 


15% 


15% 


Other 


4% 


3% 


3% 


Total 


100% 


100% 


100% 



Exhibit reads: Type of school in the CSR sample is representative of 2002 CSR universe 
and type of school in the comparison group is exactly matched to the sample. 



Exhibit B-3 

Location of All 2002 CSR Schools, CSR Sample, and Non-CSR Comparison Sample 





CSR Universe 


CSR Sample 


Comparison 


Urban 


46% 


46% 


45% 


Suburban 


23% 


20% 


23% 


Town 


13% 


12% 


10% 


Rural 


19% 


22% 


22% 


Total 


100% 


100% 


100% 



Exhibit reads: Distribution of CSR sample and comparison school by locale are 
representative of CSR universe. 



Differences exist between the CSR universe, the sample schools, and the comparison 
schools with regard to the poverty levels of the students they serve. Although the universe and 
sample of CSR schools serve similar percentages of high-poverty students (78 percent of the 
universe and 75 percent of the sample are schools with over 50 percent of students in poverty), 
the comparison schools are less likely to be high poverty (66 percent have over 50 percent of 
students in poverty) (Exhibit B-4). 
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Exhibit B-4 

Percentage of Students in Poverty at 2002 CSR Schools, CSR Sample 
and Non-CSR Comparison Sample 



CSR Universe 


CSR Sample 


Comparison 


75-100% 


45 


41 


37 


50-74% 


33 


34 


29 


24-49% 


18 


22 


26 


0-24% 


4 


3 


8 


Total 


100 


100 


100 



Exhibit reads: While distributions of the samples and CSR universe are similar, 
comparison schools were less likely to be in the highest poverty quartile. 



In order to determine baseline student performance, the team calculated z-scores for each 
CSR and comparison school. Exhibit B-5 indicates performance at baseline for the universe of 
CSR schools, the sample schools, and the comparisons schools. The CSR sample performed 
better than the universe of CSR schools, and the comparison group performed better than the 
sample at all grade levels in both reading and mathematics. 



Exhibit B-5 

Average School-Level Z-Scores (Math and Reading) 



Math 


CSR Universe 


CSR Sample 


Comparison 


Elementary 


-0.95 


-0.72 


-0.39 


Middle 


-0.86 


-0.82 


-0.38 


High 


-0.68 


-0.51 


-0.35 


Reading 


Elementary 


-0.92 


-0.77 


-0.47 


Middle 


-0.89 


-0.84 


-0.40 


High 


-0.69 


-0.52 


-0.36 



Exhibit reads: CSR universe schools had lower average baseline performance levels than 
sample and comparison schools in math and reading. 

The characteristics of the comparison schools chosen for the study are similar, but not 
exactly like the sample of CSR schools. Overall, the school level and locale of comparison 
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schools were matched closely; however, the preference for choosing comparison schools within 
the districts of the sample schools created pairs that were not ideally matched. In some districts, 
the availability of comparison schools was limited because all other CSR-eligible schools had 
already received funding, removing them from the selection pool, or the district was too small to 
have more than one school at the appropriate level. 

The comparison group for LACIO is composed of schools that have never received CSR 
funding. Their higher initial scores may reflect targeting of CSR funds to schools with greater 
need. Although, matched on prior achievement as well as demographic variables, the pool of 
available comparison schools was limited because over time the pool of potential comparison 
sites has become smaller, leaving schools that exhibit higher (although not very high) 
achievement levels. Interviews of state and district officials will explore the validity of that 
explanation. 



Replacement of Survey Sample and Comparison Schools 

Not every school in the initial sample of 400 CSR and 400 comparison schools agreed to 
participate in the evaluation. When school principals (or their districts) would not consent to 
receive surveys, they were defined as “refusing to participate.” Note that this definition excludes 
schools that received surveys yet failed to return the forms. Members of the second group were 
not dropped from the sample and were treated as nonresponders (see response rate section 
below). 

Two methods were used to replace nonparticipating schools. When comparison schools 
refused to participate, another comparison for the sample school was chosen based on the same 
criteria as the initial selection process. However, if a CSR sample school refused to participate, 
both the CSR school and matching school were removed. The new pair was chosen from the 
same locale, school level, and state to avoid biasing the sample in any systematic manner. 



Selection of Schools for Field-based Study 

With such a small number of schools in the field-based study, the sample is representative 
neither of the geographic distribution nor distribution of school levels in the universe of CSR 
schools. From the larger sample of 400 CSR and comparison school pairs, the team randomly 
selected 25 potential pairs, and narrowed the subsample further by deliberately selecting 15 
schools within the initial 25. The selection was necessary to ensure that the subsample would 
include enough geographic diversity and diversity of school levels to maximize the variety of 
policy environments available for study within the constraints of the design. The remaining list 
of 10 randomly selected schools was retained in the event that a field-based study school or 
district did not agree to be part of the evaluation. 

Researchers took an additional precaution by selecting field-based study CSR and 
comparison pairs that resided in the same district. This precaution was necessary for two reasons. 
Unlike the large-scale sample, where policy comparisons occur over the entire sample, in the 
field-based study the only way to observe the differential effect of district policies is to make 
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comparisons within a single district. Also, visiting a single district for each pair minimizes the 
data collection burden for both the evaluators and respondents. 



Selection of District Interview Sample 

The evaluation also includes school districts as part of the large-scale and field-based 
studies. The team will conduct telephone interviews with each of the 15 field-based study 
districts as well as 50 additional districts across the country. From the current sample covering 
34 states, at least one district was selected to participate per state. Districts with CSR schools 
included in the large-scale school survey were pooled within each state. Then random selections 
were made state-by- state from the districts in the pool. 



Administration of Principal and Teacher Sun’eys 

The survey instrument uses items that are evidence-based and measure behaviors, rather 
than attitudes and expectations. The use of these types of items in the survey represents our 
desire to measure respondents’ actual behaviors rather than attitudes or beliefs. 

Items asking about schools’ actual behaviors and conditions maximize the validity of 
survey data. Such items may be compared with the traditional design of survey items, which 
emphasize subjective ratings and responses from individual respondents’ and are vulnerable to 
merely soliciting “socially desirable” responses. When activity or behavioral items are used — 
e.g., “Which academic subjects are covered by goals or benchmarks for student achievement?” 
— in principle, the item is more amenable to external corroboration, and respondents are less 
likely to make a response other than by giving their most accurate one. Although the entire 
survey still consists of self-reported data, the use of such behavioral items increases the quality 
of the data (Fowler 1993). 

The research questions central to the evaluation drove the selection of survey items. 
Because the research questions focus on identifiable actions of schools and their impact as the 
result of CSR program implementation, the survey administered to participants in this evaluation 
is focused strictly on behaviors. In essence, the survey tracks behaviors that result from the 
implementation of CSR rather than attitudes about its implementation. 



Expert Review and Field Testing 

All data collection instruments were reviewed before being used in the field. Instruments 
and procedures were shared with members of the Technical Working Group, who brought their 
expertise as researchers and practitioners to their review of the design of the items, the burden on 
respondents, and the implications for data analysis. In addition to such review, the evaluation 
team pilot tested all data collection instruments, including all interview and observation 
protocols and surveys. During these tests, which were administered to no more than nine 
respondents, the team assessed item comprehension, the effectiveness of the proposed strategies 
for gaining cooperation, and the length of time for respondents to answer questions in the 
instruments. 
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Surx’ey Data Collection 

Data collection for the principal and teacher surveys took place in three stages: obtaining 
consent, distributing and collecting surveys, and follow-up. Data collection began with 
contacting the districts to inform them of the evaluation and solicit support. Most districts 
decided to administer the surveys through the superintendent's office or CSR contact person. 

This had both a positive and a negative effect. On the positive side, the district contact lent 
support to the research effort (in some cases, districts attached approval forms or letters of 
encouragement). However, it also increased the time between survey administration and 
response. 

Over the course of the data collection, schools from several districts were eliminated from 
the sample. Some districts refused to participate because of previous commitments to other 
projects. In other districts, target schools were in the process of shutting down or restructuring. 
Schools in these categories (9 sample and 16 comparison) were replaced during the study. 

Surveys were distributed to schools in three waves as consent was obtained from districts 
and schools. Follow-up began one week after mailing surveys in the form of a reminder postcard. 
After two weeks and every week thereafter, schools were contacted by phone until survey forms 
were received. Schools required an average of three to four phone call reminders before 
returning survey forms. 

In some cases, the last wave of mailing coincided with a variety of end-of-school activities. 
In other cases, districts delayed data collection until the start of school in fall 2003. For example, 
New York City officials initially agreed to participate in spring 2002 but then reversed their 
decision because a statewide research effort was underway. District officials agreed to allow us 
to administer the surveys in September. Administration and follow-up with New York City and 
other nonresponding districts (24 sample and 24 comparison schools) were concentrated in 
September. Duerr Research has continued follow-up, and groups of completed surveys continue 
to arrive at WestEd. 



Response Rate 

The data in this report are drawn from the survey forms returned to WestEd before July 30, 
2003. The 239 pairs of schools that have responded so far represent 60 percent of the original 
target of 400 pairs of CSR and comparison schools. Survey forms have come in from 318 of 367 
CSR schools surveyed so far (87 percent response rate), and 279 of 360 non-CSR comparison 
schools (78 percent response rate). However, the response rate is likely to increase as follow-up 
efforts continue, especially in the delayed and replacement schools. To ensure that we receive the 
necessary number of surveys, follow-up efforts will intensify. The evaluators are confident that 
the 88 percent response rate required will be achieved in order to get adequate power for the 
more complex statistical models that will be included in the second annual report of both sample 
and comparison schools (0.88*0.88*400=310). 

The preliminary analyses included in the report will be supplemented by a final analysis 
when data from all possible cases have been entered and processed. Further, the evaluators will 
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(once all data collection has been completed) conduct an analysis of nonrespondents to 
determine the extent to which the final sample is biased. This analysis will also appear in the 
second annual report. 



Analysis Methods — Year One 

The analysis methods used in this report are mainly descriptive with simple bivariate 
associations used when necessary. A more complex analysis including an estimation of student 
achievement outcomes will appear in the second annual report. The reason for this delay is the 
lag in availability of student achievement data corresponding to the first year of implementation 
for the 2002 cohort. 

Four sources of data were used to produce the descriptions in this report. The first source is 
the CSR Awards Database maintained by the Southwest Regional Educational Development 
Laboratory (SEDL). Lists of CSR schools are aggregated here from all U.S. states and territories. 
Second, the National Center for Education Statistics Common Core of Data was the source for 
school characteristics for CSR and comparison schools. The third source is the National School- 
Level Assessment Database maintained by the American Institutes for Research. Results of 
statewide standardized achievement tests are aggregated here. The final source of data for this 
report is the results from the administration of the LACIO Principal and Teacher Surveys. 

Summaries of school characteristics using the SEDL and NCES databases consisted of 
straightforward cross-tabulations. Schools were included in this analysis only if the NCES 
database indicated that the school was operational. 



Student Achievement Data 

Summaries of student achievement data across states were calculated in three steps. First, a 
single grade level was selected to represent the performance of a school. Often this selection was 
made simpler by the availability of only one grade level. Elementary schools were preferentially 
represented by grade 3, but if this grade was not available, grade 4 or grade 5 was selected. 

Grade 7 was used preferentially for middle schools, followed by grade 8. Grade 1 1 was preferred 
over grade 10 for high school when possible. In some cases, only an aggregate of all grades in a 
school was available to use. 

Second, because scores were reported in three different formats across states, the most 
desirable format for each state was selected. Scaled scores were used, when available, because of 
their ratio properties. Normal curve equivalent (NCE) scores, or percentile ranks converted to 
NCE, were used when scaled scores were not available because they have some interval 
properties. However, about half of the states only reported the percentage above cut scores or 
quartiles. In these cases, the distribution of each cut (often three were reported) was examined in 
order to choose the one that was closest to a normal distribution. The mean value had to be at 
least two standard deviations from the upper or lower limit of the scale. 

In the third step, the values for the selected grade level or scale were converted to z-scores 
and aggregated across states by category. This method was used in the first-year report in the 



62 




interests of simplicity. However, in future reports, student achievement from different tests and 
different reporting formats will be aggregated using meta-analytic techniques. 



Survey Data 

Survey data from teachers and principals was first tabulated for all responses on the forms. 
Because each school was mailed three teacher surveys, an intervening step was performed. 
Before tabulation, the modal response for teachers within each school was determined. This 
aggregate response was used in further analyses. 

Items on both forms were associated to check for validity. Almost all items were 
determined to be statistically significantly at the .01 level using the chi-square test, and phi was 
calculated for each item. In items with polytomous response variables, gamma was calculated to 
determine the magnitude of association. For items with continuous response variables, simple 
correlations were used. The size of the associations ranged from medium to large. Because of the 
large association between the teacher and principal responses, only the principal responses are 
reported here as the measure of school reform. However, on a small number of items minor 
disagreements between principals and teachers existed. On those items, the responses for both 
teachers and principals are both noted. These differences will be explored in subsequent reports. 

Two types of comparisons were performed on survey items. When the responses of CSR 
principals are compared with non-CSR principals, a simple chi-squared test was used. However, 
when the responses of CSR principals from items asking information about last year are 
compared with responses about this year, a McNemar chi-square was used to test the symmetry 
of rows. 
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